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FOREWORD 


The  Reliability  Management  Handbook  is  designed  to  assist  Air 
Force  System  Program  Office  (SPO)  Reliability  Engineers  in  the 
performance  of  their  assigned  management  tasks.  This  Handbook 
is  intended  to  serve  as  an  aid  in  the  implementation  of 

MEL-R-27542A  (USAF),  "Reliability  Program  for  Systems,  Subsystems, 
and  Equipment." 

It  is  recognized  that  certain  definitions  and  statements  of 
philosophy  expressed  herein  may  be  controversial.  However, 
the  Handbook  is  presented  as  an  expression  of  current  practice 
in  the  field  of  reliability  program  management.  Although  the 
Handbook  hew:  been  extensively  reviewed,  it  is  inevitable  that 
not  all  of  the  errors  and  inconsistencies  have  been  eliminated. 

In  order  to  make  this  work  available  to  you  within  a  reasonable 
time,  while  it  is  still  meaningful,  it  is  published  in  its 
present  imperfect  form.  It  is  the  intention  of  the  Aerospace 
Corporation  to  review  periodically  and  update  the  Handbook  as 
required.  This  Handbook  is  constructed  so  that  individual 
pages  or  sections  may  be  revised  as  required.  Recommendations 
for  improvement  of  the  Handbook  sure  solicited  from  users, 
interested  individuals,  and  organizations. 


REVISION  INSTRUCTIONS 


Changes  to  this  report  will  he  accomplished  hy  transmitting 
one  or  more  revision  pages.  Each  revised  page  will  be  identified 
by  the  abbreviation  Rev.  and  a  sequential  Arabic  number,  beginning 
with  1,  placed  immediately  below  the  report  number.  The  revisions 
number  reflects  the  total  number  of  revisions  to  this  report 
issue;  it  does  not  indicate  the  number  of  revisions  to  a  particular 
page. 

A  new  title  page  and  Revision  Summary  sheet  will  accompany 
each  revision.  Revision  title  pages  will  carry  the  revision 
date  below  the  revision  designation. 

Revision  Summary  sheets  (see  Page  ix)  should  be  retained  as 
a  permanent  part  of  the  particular  document  issue  to  serve  as  a 
record  of  all  change  action.  Revision  Summary  sheets  replace 
previously  issued  sheets  if  they  include  all  prior  change  infor¬ 
mation;  otherwise  they  are  handled  as  additional  pages  and 
numbered  accordingly. 
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involved  in  the  preparation  and  implementation  of  the 
particular  system  program  are  represented  in  an  integrated 
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1.2.2  Air  Force  Reliability  Coordinators 


A  Reliability  Coordinator  is  a  person  in  the  System 
Program  Office  designated  as  the  central  agent  for  relia¬ 
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1.  INTRODUCTION 


1.1  Importance  of  Reliability 
In  Military  Programs 


The  field  of  reliability  engineering  has  grown  in  the  last 
decade  frcm  a  limited  study  of  electron  tube  failures  to  an 
engineering  discipline  of  considerable  stature.  What  was  formerly 
a  highly  specialized  area  of  consideration  is  today  a  formal  and 
systematic  branch  of  technology.  Numerous  factors  have  contributed 
to  the  tremendous  growth  of  the  reliability  field.  But,  paramount 
among  them  is  the  insistence  by  the  military  that  contractors 
actively  practice  the  principles  and  techniques  of  reliability 
engineering . 

The  achievement  of  desired  results  in  a  reliability  program 
is  closely  related  to  a  customer's  acceptance  of  a  given  degree 
of  risk  in  respect  to  end-product  performance.  Time  and  resources 
expended  will  determine  this  degree  of  risk.  It  follows,  then, 
that  one  of  the  initial  and  major  challenges  which  confront  a 
procuring  activity  is  the  determination  of  expenditures  to  be 
allotted  to  a  program.  The  importance  of  this  challenge  —  as  it 
affects  ultimate  product  capability  --  demands  that  reliability 
be.  considered  as  a  system  parameter  and  that  this  parameter  be 
designated  as  the  specific  responsibility  of  seme  person(s) 
within  the  organizational  structure  for  systems  management.  In 
Air  Force  systems  management,  the  responsibility  would  logically 
be  assigned  to  the  System  Program  Office  (SFO)  Reliability 
Coordinator . 

Accordingly,  with  the  recognition  of  the  proper  consideration 
of  reliability  as  a  vital  factor  in  the  planning,  design,  develop¬ 
ment  and  testing  of  complex  military  systems,  it  is  Important  to 
provide  suitable  guides  to  management  concerning  the  scope  and 
application  of  reliability  engineering  concepts  and  techniques 
in  systems  engineering.  A  description  of  the  system  engineering 
process  is  presented  in  Section  1.5  Including  an  explanation  of 
the  role  of  reliability  engineering.  This  Reliability  Management 
Handbook  has  been  prepared  to  increase  the  understanding  of  System 
Program  Office  personnel  with  systems  reliability  considerations 
and  how  they  interface  with  other  systems  characteristics,  such  as 
performance,  maintainability,  operability,  effectiveness,  availa¬ 
bility,  safety,  supportab&llty ,  procnrabllity,  produclbility  and 
cost. 


1.2  Air  Force  Organization  for 
Reliability  Management 


1.2.1  Systems  Management  Concept 


The  organizational  structure  of  the  Air  Force  Is 
characterized  by  two  distinct  categories  of  management 
activity: 

(1)  Functional  Management  - - the  grouping  of 

responsibilities  according  to  the  type  of 
work  to  be  performed,  that  is,  plans, 
operations,  research  and  development, 
procurement  and  production,  supply  and 
maintenance,  personnel  and  comptroller. 

(2)  Systems  Management  --  the  process  of  organ¬ 
izing  and  employing  functional  agencies  to 
accomplish  objectives  of  a  clearly  defined 
weapon,  support,  or  command  and  control 
system  program. 

The  systems  management  concept  requires  establishment 
of  a  separate  System  Program  Office  (SPO)  for  each  approved 
system.  This  office  provides  a  management  focal  point  and 
central  locale  where  the  Air  Force  functional  agencies 
involved  in  the  preparation  and  implementation  of  the 
particular  system  program  are  represented  in  an  integrated 
organization. 

1.2.2  Air  Force  Reliability  Coordinators 


A  Reliability  Coordinator  is  a  person  in  the  System 
Program  Office  designated  as  the  central  agent  for  relia¬ 
bility  policy,  plans,  programs,  and  activities.  As  pre¬ 
scribed  in  AFSCR  80-1,  Reliability  Program  for  Aerospace, 
Support,  and  Command  and'  Control  Systems,  each  SPO  will 
"maintain  an  adequate  staff  for  monitoring  and  guidance 
of  the  reliability  program  for  the  system  with  which  they 
are  concerned."  Consistent  with  the  systems  approach  to 
the  management  task,  the  functions  and  responsibilities 
of  the  SPO  Reliability  Coordinator  can  thus  be  assumed  to 
snoompass  all  matters  which  contribute  to  or  in  any  way 
affect  the  reliability  of  the  system. 


1.3  Purpose  and  Content  of  Reliability 
Management  Handbook 


This  handbook  has  been  compiled  to  serve  as  a  guide 
to  SPO  Reliability  Coordinators  In  the  performance  of 
their  assigned  management  tasks.  In  accordance  with 
this  objective,  the  handbook  is  oriented  toward: 

(1)  Defining  the  over-all  task  of  reliability  man¬ 
agement  by  dividing  it  into  specific  functions. 

(2)  Categorizing  existing  Air  Force  policies  and 
responsibility  assignments  in  respect  to  these 
management  functions. 

(3)  Suggesting  procedures  for  implementing  the 
functions . 

(4)  Serving  as  an  aid  in  the  Implementation  of 
MIL-R-27542A(USAF),  "Reliability  Program 
for  Systems,  Subsystems,  and  Equipment." 


The  guidance  Information  contained  herein  is  not  a 
substitute  for  sound  management  practice  or  managerial 
ability  (these  factors  have  no  substitute).  Rather,  it 
is  intended  to  contribute  to  the  positive  approach  neces- 
sare  for  the  achievement  of  the  objectives  which  govern 
a  reliability  program.  It  augments  the  training,  expe¬ 
rience,  and  Judgment  of  the  SPO  Reliability  Coordinators. 

Sections  2,  3,  and  4  of  the  handbook  are  organized  to 
reflect  the  three  major  functions  which  evolve  from  the 
reliability  management  task,  as  illustrated  in  Figure  1-1: 

FIGURE  1-1 

MAJOR  FUNCTIONS  WITHIN 
OVER-ALL  RELIABILITY  MANAGEMENT  TASK 


Within  the  appropriate  section,  the  functions  of  program 
establishment,  administration,  and  surveillance  and  re¬ 
view  are  further  subdivided  into  detailed  management 
activities.  Definitions,  pertinent  policy  and  responsi¬ 
bility  statements  from  Air  Force  directives,  suggested 
implementation  procedures  and  techniques,  and  listings 
of  additional  reference  reading  are  Included  in  these 
activity  discussions. 

Sections  2,  3*  and  k  are  primarily  concerned  with 
answers  to  the  question,  HHow  does  the  SFO  Reliability 
Coordinator  manage  a  reliability  program?"  Answers  to 
the  question,  "What  does  he  manage?",  are  provided  by 
Appendix  A,  Ilements  of  a  Reliability  Program.  This 
appendix  defines,  describes,  and  discusses  the  disciplines 
and  activities  which  cumulatively  constitute  a  contractor's 
reliability  program  as  required  by  MIL-R-27542A. 


0 


1.4  Guides  for  Utilizing  Handbook  Material 


0 


The  scope  of  material  presented  in  Sections  2,  3» 
and  4  and  the  appendices  of  the  handbook  is  intentionally 
broad  in  order  to  provide  as  comprehensive  coverage  of 
the  reliability  management  task  as  possible.  The  follow¬ 
ing  general  comments  and  guidance  discussions  are  in¬ 
cluded  to  suggest  effective  ways  of  extracting  and  applying 
this  wide  range  of  information: 

(1)  Reliability  Management  Functions  and  Activities 
—  The  discussions  in  Sections  2,  3,  and  4  are 
based  on  a  descriptive  outline  or  breakdown  of 
the  over-all  task  of  reliability  management 
(this  outline  is  apparent  from  the  table  of 
contents  for  these  sections).  Specific  attempts 
have  been  made  to  highlight  the  major  management 
functions  and  activities  which  confront  the 
System  Program  Offices  and  SPO  Reliability  Co¬ 
ordinators.  Hence,  such  topics  as  "Preparation 
of  Statement  of  Reliability  Requirements"  and 
"Evaluation  of  Proposals"  have  been  selected 

as  subjects  for  discussion. 

(2)  Air  Force  Reliability  Documents  —  Several  forms 
of  documentation  relating  to  reliability  have 
been  issued  by  all  command  levels  within  the  Air 
Force.  These  documents  represent  a  prime  source 
of  authority  and  direction  for  work  performed  by 
SPO  Reliability  Coordinators.  Three  significant 
regulations  have  been  singled  out  for  special 
consideration  within  this  handbook: 


(a)  Air  Force  Regulation  No.  80-5,  Reliability 
Program  for  Systems,  Subsystems,  and 
Equipments,  4  June  1962.  1 


(b)  Air  Force  Systems  Command  Regulation  No.  80-1, 
Reliability  Program  for  Aerospace,  Support, 
and  Command  and  Control  Systems,  i4  pec.  io6g. 


(o)  Air  Force  Ballistics  Systems  Division 

Regulation  No.  80-5,  Reliability  Program 
Management ,  28  Dec .  1962. 


Appendix  B  presents  the  entire  contents  of  the 
above  listed  regulations  in  tabular  form,  arranged 
by  similar  subject  categories. 


1-5 


(3)  Additional  Reading  Materials  —  Several  published 
references  have  been  listed  throughout  the  hand¬ 
book  to  inform  the  SPO  Reliability  Coordinators 
of  the  availability  of  additional  treatments  of 
the  subjects  covered.  The  following  publica¬ 
tions  are  also  pointed  out  as  general  informa¬ 
tion  sources  of  significant  interest  to  those 

in  the  reliability  management  field: 

(a)  Proceedings  of  the  Annual  National  Symposium 
sponsored  by  the  IRE  Professional  Group 

on  Reliability  and  Quality  Control. 

(b)  Published  papers  of  the  Military- Indus try 
Missile  and  Space  Reliability  Symposium 
sponsored  by  the  Office  of  the  Director 
of  Defense  Research  and  Engineering. 

(4)  Reliability  Management  Procedures  —  Where  pos¬ 
sible,  specific  procedures  are  suggested  for 
implementing  reliability  management  activities. 

For  example,  in  discussing  the  subject  of  evalua¬ 
tion  of  contractor  status  reports,  a  check  list 
of  sixteen  items  (see  Table  3-2)  has  been  pre¬ 
sented  as  a  model  with  specific  application. 
Procedural  examples  presented  in  this  handbook 
are  not  intended  to  convey  that  stereotyped 
methods  exist  for  accomplishing  the  given  tasks. 
Rather  they  are  intended  to  suggest  approaches 
and  to  encourage  the  individual  SPO  Reliability 
Coordinators  to  develop  techniques  on  the  basis 

of  their  own  sets  of  program  needs  and  objectives. 

(5)  Reliability  Management  Policies  and  Responsibilities 
—  A  major  responsibility  of  any  Air  Force  manager 
Is  to  carry  out  the  directives  established  by  higher 
authority.  Hence,  It  Is  felt  that  prevailing 
pbllcy  statements  and  responsibility  assignments 
constitute  a  major  frame  of  reference  for  the  per¬ 
formance  of  reliability  management  functions.  In 
this  handbook,  each  major  topic  of  discussion  Is 
prefaced  by  a  tabular  presentation  of  appropriate 
policy  and  responsibility  statements. 

(6)  Relation  of  Reliability  to  Other  System  Parameters 
--  In  utilizing  the  material  contained  in  this  . 
handbook,  the  reader  should  understand  that 
"reliability"  Is  but  one  of  the  major  elements 
which  contribute  to  the  performance  capability 

of  a  ays tea.  Other  important  system  parameters 


) 
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include  maintainability,  availability,  and  design 
adequacy.  All  of  these  parameters  are  interrelated 
in  the  broader  concept  termed  "system  effectiveness." 
Appendix  P  has  been  included  to  depict  the  relation¬ 
ships  between  reliability  and  other  system  attributes 
and  to  define  the  terms  of  interest  which  are  normally 
used  in  describing  these  relationships. 

(7)  Aids  to  Location  of  Subject  Material  —  There  are 
certain  characteristics  of  this  handbook  which  are 
specially  intended  to  assist  the  user  in  locating 
material  relating  to  a  particular  subject.  These 
include  the  subject  matter  index  (at  end  of  handbook), 
the  table  of  contents,  the  MIL- R-2J^kSk  cross-reference 
chart  (page  1-9,  following,  and  page  A-2  of  Appendix  A), 
and  parenthetical  references  to  other  appropriate 
paragraphs  throughout  the  text. 
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The  material  presented  In  Appendix  A  has  been  based, 
to  the  extent  possible,  on  the  philosophy  and  specifica¬ 
tions  expressed  in  MIL-R-275^2A,  Reliability  Program  for 
Systems,  Subsystems,  and  Equipment,  dated  21  May 
(however,  it  Is  not  limited  in  that  respect).  The  extent 
of  this  correlation  has  been  depicted  by  the  table  shown 
on  the  following  page.  A  0  Indicates  applicability  of 
sections  in  Appendix  A  to  specific  paragraphs  in  MIL-R- 
27 542 A.  The  table  has  been  included  herein  to  guide  the 
Reliability  Coordinator  in  the  use  of  the  Appendix  A 
material . 
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1.5  The  System  Engineering  Process 


Rapid  advancement  of  technology  in  recent  years  has  made  possible  the 
design  and  development  of  increasingly  complex  military  systems.  Depend¬ 
ability  of  weapons  to  accomplish  their  role  in  the  destruction  of  an 
enemy’s  capability  to  overrun  and  destroy  the  homeland  has  always  been 
the  goal  of  military  equipment  engineers.  Simple  weapons  systems  such 
as  the  bow  and  arrow,  the  rifle,  and  the  cannon  could  be  invented,  produced, 
and  tested  by  a  few  craftsmen  under  the  personal  direction  and  supervision 
of  an  individual  engineer  or  master  craftsman.  They  could  be  easily 
tested  before  their  mass  duplication  in  order  to  prove  their  reliability 
under  operational  and  environmental  conditions  such  as  were  to  be  faced 
on  prospective  battlefields.  Through  World  War  II,  the  airplane  and  its 
manually  operated  machine  guns  and  bomb  sights  could  be  designed,  developed 
and  tested  by  specialists  groups,  still  largely  under  the  personal  direction 
and  surveillance  of  a  chief  engineer.  The  weapons  system  was  then 
integrated  and  their  operational  worth  demonstrated  by  flight  test 
engineerng  crews.  The  normal  development  cycle  took  such  a  weapons  system 
progressively  and  serially  through  experimental,  prototype  and  production 
models.  The  requirement  to  demonstrate  the  reliability  of  new  weapon 
systems  has,  therefore,  always  existed.  A  tactical  advantage  has  always 
been  gained  by  the  combat  force  which  possessed  the  operationally  dependable 
weapons.  Until  about  1950-  this  quality  could  be  most  easily  and  directly 
achieved  by  individual  engineers  and  small  groups  of  ordnance  specialists. 

It  was  inherent  in  their  engineering  process. 

With  the  advent  of  scientific  and  technological  breakthroughs  which 
increased  the  effective  speed,  range,  accuracy  and  destructive  power  of 
weapons  systems,  the  need  arose  for  reliability  engineering  as  a  specialty. 
The  reasons  that  it  has  become  a  specialty  rather  than  a  simple  quality 
that  an  inventor  or  engineer  seeks  in  designing,  developing  and  proving 
out  a  new  weapon  are  several: 

a.  No  one  individual  is  any  longer  capable  of  possessing  sufficient 
knowledge  covering  all  of  the  scientific  and  technical  fields 
which  80*6  employed  in  order  to  solely  make  the  necessary 
technical  Judgments  required  to  develop  an  effective  and  dependable 
military  system. 

b.  To  integrate  the  efforts  of  the  large  number  of  varied  technical 
specialists  in  designing  and  developing  components  for  incorporation 
into  a  system  requires  a  well-defined  process  and  a  communication 
system  to  produce  a  total  system  design. 

c.  The  time  period  permitted  for  developing  and  deploying  new  complex 
military  systems  is  necessarily  compressed  for  strategic  reasons. 

d.  Cost  of  developing  and  procuring  a  new  system  has  multiplied 
manifold,  reducing  the  national  capability  to  support  parallel 
competing  systems  development  in  order  to  Insure  a  desired 
capability. 
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Because  of  these  considerations,  there  has  been  an  emerging  awareness 
of  the  need  for  and  the  importance  of  a  discipline  which  will  bring  about 
an  integration  of  specialists '•  efforts  to  achieve  desired  system  outputs. 
Groups  of  specialists,  therefore,  have  emerged  as  adjuncts  to  the  design, 
development  and  testing  programs  for  complex  military  systems .  Each  in 
their  own  way  has  attempted  to  supply  an  information  processing  method 
and  technique  for  identifying  and  interrelating  the  various  qualitative 
and  quantitative  aspects  of  systems  development.  Each  specialist  group 
has  found  it  necessary  to  make  such  an  attempt  in  order  to  properly 
achieve  its  desired  quality  or  characteristic  in  a  system's  output. 

These  engineering  specialist  groups  have  been  concerned  with  achieving 
such  things  as:  reliability;  maintainability;  operability;  supportability; 
invulnerability;  safety;  producibility;  procurability;  and,  other  "ilities." 
These  groups  of  specialists  which  emphasize  the  system  design  approach 
may  be  identified  in  organizations  under  such  terms  as:  Reliability; 
Maintainability;  Facilities;  Ground  Support;  Safety;  Human  Engineering; 
System  Integration  and  Testing;  Functional  and  Task  Analysis;  Cost 
Analysis;  and,  others.  All  such  groups  have  the  same  characteristic  of 
not  being  specifically  tied  to  the  engineering  of  a  particular  kind  of 
equipment. 

The  experience  of  each  of  these  specialty  groups  in  trying  to  furnish 
the  requirements  and  to  be  the  forcing  function  upon  the  equipment  oriented 
engineering  groups  for  designing  end  items  on  an  integrated  systems  basis 
has  been  frustrating;  and ,  their  goals  for  attaining  desired  levels  of 
systems  effectiveness  have  fallen  far  short  of  being  realized.  However, 
their  efforts  and  their  insistent  promotion  of  the  systems  viewpoint, 
along  with  accumulated  experience  gained  in  the  operational  deployment  of 
complex  weapons  systems,  have  forced  responsible  systems  management 
personnel  to  realize  that  a  system  must  be  designed  and  tested  as  a 
complete  entity. 

The  word  "System,"  therefore,  has  come  through  actual  practice  to 
include:  prime  mission  equipment;  the  facilities  required  to  operate 
and  maintain  the  system;  the  selection  and  training  of  specialist  personnel; 
software;  the  operational  and  maintenance  procedures;  Instrumentation  and 
data  reduction  for  test  and  evaluation;  special  activation  and  acceptance 
progrems;  and,  logistics  support  programs  for  spares  and  depot  maintenance. 

All  parte  of  a  system  must  have  a  common  unified  purpose :  to  contribute 

to  the  production  of  a  single  set  of  optiimmi  outputs  from  given  input(s) 
with  respect  to  some  appropriate  measure  of  effectiveness.  Increasing 
recognition  of  this  fundamental  requirement  by  responsible  systems  manage¬ 
ment  personnel,  is  leading  to  the  promulgation  of  policy  directives  from 
the  Office  of  the  Secretary  of  Defense;  Regulations  and  Directives  from 
Headquarters  USAF;  Regulations  and  Manuals  from  Air  Force  Systems  Command; 
and.  revision  of  Specifications  and  Exhibits  at  Systems  Divisions  and 
Systems  Programs  Office.  These  will  serve  to  Implement  and  contract  for 
a  coarplete  systems  engineering  effort  in  the  study,  design,  development 
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and  testing  of  an  aerospace  weapon,  support  or  space  system.  The  term 
"Systems  Engineering"  has  been  accepted  and  Is  being  used  to  encompass 
and/or  replace  such  terms  as:  systems  thinking,  systems  approach,  systems 
synthesis,  systems  analysis,,  functional  analysis,*  task  analysis,  system 
definition,  system  concept  and,  team  development  method. 

System  engineering  is  fundamentally  concerned  with, deriving  a 
coherent  system  design  to  achieve  stated  objectives.  The  system 
engineering  process  logically  considers  and  evaluates  each  of  the 
innumerable  military,  technical  and  economic  variables  identified  by  the 
system  engineers.  Choices  of  methods  of  system  operation  and  the  system 
elements  is  a  highly  involved  process ,  for  a  change  in  one  system  variable 
will  affect  many  other  system  variables,  rarely  in -a  linear  fashion.  The 
generation  of  a  well-balanced  system  design  requires  that  each  major  system 
decision  be  based  upon  the  proper  consideration  of  other  system  variables 
such  as  cost,  facilities,  personnel  requirements,  procedural  data  require¬ 
ments,  testing  and  logistics.  Further,  it  requires  a  balancing  among  the 
considerations  leading  to  incorpoation  of  design  characteristics  which 
will  produce  reliability,  maintainability,  operability,  safety,  and 
support ability.  To  achieve  desired  system  performance  effectiveness  and 
dependability  requires  the  closest  coordination  amongst  system  engineers 
and  implies  a  design  team  of  select  specialists  skilled  in  system  engineering. 
This  team  has  the  responsibility  of  translating  military  operational  or  advanced 
developmental  requirements  into  a  feasible,  economical  system.  This  team 
responsibility  will  not  be  satisfied  until  system  tests/demonBtrations 
have  proven  the  adequacy  of  the  production  end-item  specifications, 
personnel  selection  and  training  data,  facilities  and  procedural  data. 

Each  of  the  members  of  this  system  engineering  team  will  have  a 
different  and  probably  unique  educational  and  technical  experience 
background.  They  must  represent  or  be  thoroughly  knowledgeable  concerning 
each  technical  specialty  to  be  employed  in  planning,  designing,  developing 
and  testing  a  given  system.  No  two  systems  are  ever  alike  in  their 
development  requirements.  However,  the  process  for  arriving  at  logical 
system  decisions  is  Identical,  regardless  of  system  purpose,  size  or 
complexity.  Regardless  of  their  particular  technical  specialty,  therefore, 
system  engineers  must  be  trained  and  given  experience  in  system  engineering, 
and  be  devoted  to  arriving  at  the  best  system  design,  considering  all  of 
the  requirements  and  constraints  to  be  met. 

Reliability  engineering  and  reliability  engineers  play  an  important 
and  unique  role  in  this  system  engineering  process. 

First,  some  reliability  engineers  Bust  become  qualified  systems 
engineers  and  participate  in  the  system  engineering  team  effort  in 
arriving  at  the  system  design. 
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Second,  all  reliability  engineers  must  be  fully  informed  and  skilled 
in  the  application  of  reliability  engineering  methods  and  techniques  in 
executing  the  detailed  design,  development  and.  testing  of  equipment  end- 
items.  The  reliability  engineers  must  be  famaliar  with  the  facilities 
and  environmental  controls,  the  operations  and  maintenance  procedures, 
including  personnel  performance,  and  the  systems  testing  and  deployment 
of  these  end-items.  This  background  will  enable  the  reliability  engineer 
to  participate  in  the  systems  engineering  activity. 

Third,  some  reliability  engineers  will  be  highly  specialized  sind 
skilled  in  the  application  and  development  of  methods  and  techniques 
to  be  employed  by  other  reliability  engineers.  These  applied  research 
activities  will  contribute  to  the  analysis  and  evaluation  processes  for 
obtaining,  deriving,  reducing,  and  evaluating  assumed  or  obtained  equip¬ 
ment  and  system  performance  data  for  the  purpose  of  making  Judgments 
about  the  probable  dependability  of  the  system  for  successfully  achieving 
its  performance  objectives. 

In  this  connection,  the  achievement  of  desired  results  by  a  system 
from  the  viewpoint  of  reliability  considerations,  must  be  related  to  a 
customer's  acceptance  of  a  given  degree  of  risk  in  respect  to  its  end- 
product  performance.  Time  and  resources  (both  material  and  human)  expended 
will  establish  this  degree  of  risk.  It  follows,  then,  that  one  of  the 
initial  and  major  challenges  which  confront  a  procuring  activity,  such 
as  the  Office  of  the  Secretary  of  Defense,  and  its  agent  the  Air  Force 
Systems  Command,  is  the  determination  of  the  expenditures  to  be  allocated 
to  a  program.  In  helping  to  arrive  at  a  system  design,  therefore,  the 
reliability  engineer  functioning  in  the  role  of  a  systems  engineer  must 
objectively  accept  this  requirement  and  contribute  his  findings  as 
"best  Judgments,"  and  to  honestly  state  the  most  probable  outcomes  of 
incorporating  given  design  characteristics.  It  1b  for  the  lack  of  this 
skill  which  can  be  applied  by  a  well-qualified  reliability  engineer, 
trained  and  experienced  in  the  system  engineering  process,  that  has  led 
in  the  past  to  costly  overruns  and  schedule  slippages  on  some  systems, 
or  where  degraded  capability  has  been  accepted  in  order  to  achieve  some 
performance  value  out  of  an  expensive  and  tactically  important  system. 

Because  of  complexity  and  costs,  war  plans  have  been  made  dependent  upon 
a  specified  system  performance  capability  becoming  available  for  a  given 
time  period.  For  the  lack  of  proper  and  accurate  reliability  consider¬ 
ations,  at  the  right  time  and  in  the  proper  context  during  system  engineering, 
the  national  security  can  be  seriously,  and  perhaps  fatally  Jeopardized. 

It  is  for  this  most  urgent  reason,  and  the  examples  of  failures  in 
effective  systems  engineering  in  the  past,  that  system  management  is 
currently  concerned  with  achieving  the  objectives  described  in  this 
handbook  as  one  of  the  Important  attributes  and  contributions  to  effective 
systems  engineering. 
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2.  ESTABLISHMENT  OP  A  RELIABILITY  PROGRAM 


Establishment  of  an  effective  reliability  program  Is 
a  task  which  Is  essential  to  the  ultimate  achievement  of 
desired  system  reliability  In  a  contracted  program.  It 
Is  a  task  which  places  major  responsibilities  upon  the 
customer  as  well  as  the  contractor. 

The  customer  must  lay  the  basic  foundation  which  will 
assure  eventual  accomplishment  of  a  design  by  equipment 
manufacturers.  He  must  formulate  an  adequate  and  realistic 
statement  of  reliability  requirements  for  the  system  (2.1); 
he  must  select  the  most  qualified  contractor  (or  contrac¬ 
tors)  for  fulfilling  these  requirements  (2.2);  and  he  must 
specify  and  evaluate  the  reliability  activities  (relia¬ 
bility  program  plan)  to  be  implemented  by  the  contractor 
(2.3). 


Frequent  occasions  have  arisen  in  which  the  efforts 
of  a  contractor  to  achieve  desired  reliability  in  a  system 
have  proven  to  be  inadequate.  In  analyzing  the  cause  of 
the  inadequacy,  reference  must  invariably  be  made  to  the 
initial  statement  of  reliability  requirements  provided  by 
the  customer.  Common  complaints  registered  by  contractors 
or  their  representatives  have  included  the  following: 

(1)  Program  requirements  as  contractually  stated 
were  lacking  in  sufficient  detail,  ambiguous, 
or  not  applicable  to  the  specific  product  in 
question . 

(2)  Overlapping,  inconsistent,  or  conflicting  re¬ 
quirements  were  created  by  a  complex  contract 
structure  incorporating  several  different 
specifications,  standards,  and  other  oblige-, 
tory  statements. 

(3)  The  program  requirements  were  unnecessarily 
demanding . 

(4)  Requirements  which  altered  the  scope  of  work 
were  imposed  on  the  oontraotor  subsequent  to 
contract  negotiation. 

(5)  Requirements  were  imposed  on  the  oontraotor 
on  an  unofficial  basis,  that  Is,  without 
stipulation  in  the  oontraot. 


Such  problems  can  never  be  entirely  eliminated  from  programs 
which  are  complex  in  scope  and  requirements.  They  can  be 
minimized,  however,  through  conscientious  and  detailed  attention 
to  the  disciplines  and  tasks  necessary  to  establishment  of  an 
effective  reliability  program. 

Section  2  of  this  handbook  provides  guidance  to  SPO  Relia¬ 
bility  Coordinators  in  determining  a  deliberate  and  definitive 
course  of  action  relative  to  establishment  of  a  reliability 
program.  The  functions  and  responsibilities  described  in  this 
section  are  those  which  are  properly  the  responsibility  of  a  SPO 
Reliability  Coordinator  (whether  he  performs  the  tasks  himself 
or  directs  the  performance  of  others).  They  encompass  a  time 
period  which  may  be  considered  to  extend  from  the  first  decision 
of  the  customer  to  recognize  reliability  as  a  factor  of  interest 
in  a  system  program  to  the  official  approval  of  a  contractor's 
reliability  program  plan  by  the  procuring  activity  (usually  at 
some  time  immediately  following  the  negotiation  of  a  contract). 
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2.1  Formulation  of  Reliability  Requirements 


There  are  two  basic  approaches  which  can  be  utilized 
In  formulating  reliability  requirements  for  a  contracted 
program : 

(l)  Prescribing  reliability  requirements  in,  de¬ 
tail  not  only  for  the  system  but  also  for 
components,  materials,  and  processes  so  that 
all  elements  of  work  and  levels  of  effort  in 
the  program  are  controlled  as  much  as  possible. 
This  method  is  used  to  provide  added  assurance 
to  the  customer  that  end  items  delivered  after 
acceptance  of  the  product  will  continue  to 
meet  requirements,  since  controls  are  Imposed 
on  all  aspects  of  product  production  (e.g., 
quality  of  materials  and  components  must  be 
continuously  tested  and/or  inspected). 


(2)  Prescribing  reliability  requirements  only  for 
the  end  product.  This  process  allows  a  minimum 
'  of  Customer  interference  and  provides  the  con¬ 
tractor  complete  freedom  in  determining  and 
implementing  the  methods  to  be  used  in  achieving 
the  reliability  requirements.  Determination  of 
reliability  achievement  is  accomplished  through 
a  demonstration  program  conducted  on  the  cus¬ 
tomer's  behalf. 

In  formulating  the  reliability  requirements  for  a 
specific  program,  usually  a  compromise  set  of  requirements 
will  evolve,  including  some  detail  requirements  as  well  as 
end-product  requirements.  The  extent  to  which  detail  re¬ 
quirements  are  specified  will  also  depend  on  parameters 
other  than  reliability,  suoh  as  the  time  scale  and  cost 
of.  the  program. 

The  formulation  of  program  reliability  requirements 
is,  hypothetically,  independent  of  and  preliminary  to  the 
selection  of  a  contractor.  Activity  to  establish  such 
requirements  commences  at  the  time  of  decision  to  in¬ 
corporate  reliability  into  a  program  and  continues  until 
the  program  requirements  are  stated  in  documentary  form. 
This  documentary  form  is  usually  a  contractual  "Statement 
of  Work,"  although  in  certain  cases  it  may  be  an  exhibit, 
a  specification,  or  a  standard.  In  any  event,  it  repre¬ 
sents  the  cumulative  statement  of  the  customer- imposed 
requirements  that  must  be  considered  by  prospective  con¬ 
tractors  in  Submitting  bids  for  performing  work. 


2.1.1 


Principal  sources  of  directives  which  dictate  Air 
Force  policy  in  respect  to  requirements  formulation  are: 

(1)  Air  Force  Regulation  80-5  (AFR  80-5) >  ^  Jua*  1962 

(2)  Air  Force  Systems  Command  Regulation  80-l>  lA  Dec.  1962 
(AFSCR  80-1) 

(3)  Air  Force  Ballistic  Systems  Division  Regulation 

80-5  (AFBSDR  80-5),  28  Dec.  1962 

All  of  these  specify  that  reliability  is  a  factor  that 
must  be  considered  during  program  planning  stages  and 
must  embrace  both  quantitative  and  qualitative  aspects. 

Further,  they  vest  direct  responsibility  for  formulation 
of  reliability  requirements  In  the  System  Program  Office 
(SPo),  and  thus  in  the  SPO  Reliability  Coordinator. 

In  the  discussion  of  2.1,  the  task  of  reliability 
requirements  formulation  is  subdivided  into  three  areas 
of  consideration: 

(1)  Selection  of  quantitative  requirements  (i.e., 
the  numerical  statement  of  required  relia¬ 
bility)  . 

(2)  Selection  of  qualitative  requirements  (i.e., 
the  elements  of  work  to  be  Implemented  In  a 
reliability  program) . 

(3)  Statement  of  oontraot  requirements. 

These  sub tasks  are  Interrelated  in  many  respects,  but 
each  also  has  specific  needs  and  requires  specific  con¬ 
siderations  and  documentation. 

Tasks  (l)  and  (2)  above  suggest  the  performance  of 
analytical  studies,  consideration  of  trade-offs,  and 
formulation  of  decisions  which  are  Intended  to  Identify, 
describe,  or  otherwise  depict  the  objectives  and  require¬ 
ments  of  the  reliability  program  for  a  defined  system. 

After  these  tasks  have  been  essentially  completed.  It  Is 
then  the  primary  conoem  of  the  SPO  Reliability  Coordinator 
to  dooument  the  program  requirements  In  a  statement  form 
to  be  used  by  the  procuring  activity  In  selecting  a  con¬ 
tractor  to  perform  the  work. 


2.1.1  Selection  of  Quantitative 
Reliability  Requirements 

2. 1.1.1  Definition 


Reliability  la  a  system  parameter  in  that  It  Is  con¬ 
trollable  and  can  be  measured  under  specific  conditions 
of  operation.  This  concept  of  quantitative  reliability 
Is  recognized  within  existing  Air  Force  philosophy  and  is 
manifest  in  directives  from  all  command  levels.  Such 
directives  stipulate  that  quantitative  reliability  require¬ 
ments  be  considered  during  the  conceptual  phases  of  planned 
programs . 

The  activity  discussed  In  2.1.1  Is  defined  as  that 
effort  which  relates  to  the  determinations ,  decisions, 
and  stipulations  instrumental  to  selection  of  quantita¬ 
tive  reliability  requirements,  nils  activity  commences 
at  the  time  of  program  identification  and  continues  until 
a  statement  of  quantitative  requirements  is  defined. 


2. 1.1. 2  Air  Force  Policy 


Air  Foroe  policy  which  relates  to  selection  of  quan¬ 
titative  reliability  requirements  Is  summarized  in  Table 
2-1. 


2. 1.1. 3  Responsibilities  for  Quantitative 
Requirements  Selection 


The  responsibilities  of  System  Program  Offices  (and 
thus  SPO  Reliability  Coordinators)  and  other  associated 
levels  of  authority  relating  to  selection  of  quantitative 
requirements  are  summarized  In  Table  2-2. 
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TABLE  2-1 


SUMMARY  OP  AIR  FORCE  POLICY 


RELATING  TO  SELECTION  OP  QUANTITATIVE  RELIABILITY  REQUIREMENTS 


Air  Force 
Document 

Statement  of  Policy 

Par. 

No. 

APR  80-5  . 

4  June  1962 

(1)  During  the  conceptual  and  acquisition  phases, 
systems  will  be  analyzed,  and  a  reliability  program 
established  for  each.  Each  program  will  include  a 
minimum  acceptable  reliability  level  as  well  as  a 
reliability  goal,  with  intermediate  quantitative 
values  required  to  measure  progression.  Where  suf¬ 
ficient  sampling  permits,  a  stated  minimum  accept¬ 
able  confidence  level  should  be  included  for  each 
probability  value. 

4.a.(3) 

(2)  Reprocured  spares  and  parts  will  be  procured  to 
definitive  reliability  requirements. 

4. a. (4) 

(3)  Specifications,  exhibits,  work  statements,  pro¬ 
duct  descriptions,  and  contracts  for  systems  and 
associated  materiel,  including  GPE  for  inventory, 
will  include  specific  minimum  acceptable  reliability 
requirements  as  one  of  the  major  engineering  factors. 
Individual  parts  specifications  will  include  current 
failure  rate  level  and  up  to  four  discrete  graduated 
levels  representing  state. of  the  art  advancements. 

The  total  number  of  levels  will  be  governed  by  sys¬ 
tems  requirements. 

4.b 

(4)  System  contracts  will  include  a  requirement  for 
a  comprehensive  contractor  reliability  program,  in¬ 
cluding  quantitative  requirements. 

4.  c 

AFSCR  80-1 

14  Dec.  1962 

(1)  Proposed  system  package  plans  (PSPP's),  system 
package  programs  (SPP's),  and  development  plans 

TdP'b;  will  contain  a  narrative  statement  delineat¬ 
ing  the  desired  reliability  characteristics.  How¬ 
ever,  comprehensive  reliability  programs  for  feasi-  . 
bllity  studies,  exploratory  development,  and 
advanoed  development  categories  are  not  desired.. 

4.b 

(2)  Quantitative  reliability  requirements  will  be 
developed  from  stated  objectives  and  stated  in  speci¬ 
fic  numerical  terms  in  the  appropriate  contractual  ' 
documents  and  systems  specifications  and  will  include 
the  definition  of  satisfactory  operation,  including 
operating  time  or  cycles,  environmental  conditions, 
and  where  practicable,  the  confidence  levels  to  which 
the  specified  probabilities  will  be  demonstrated. 

4.c 

(3)  Quantitative  reliability  figures  will  be  stipu¬ 
lated  for  the  reliability  goal,  the  minimum  accept¬ 
able  reliability  requirements  and  the  achieved 
reliability  at  such  specified  intermediate  points 
as  neoessary  to  measure  progress. 

4.d 

APBSDIl  80-5 

28  Dec.  1962 

(1)  Quantitative  requirements'  and  provisions  for 
demonstration  will  be  incorporated  into  future 
contracts  and  into  existing  contracts  as  appro¬ 
priate. 

5- a 
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TABLE  2-2 


SUMMARY  07  RESPONSIBILITIES 
REIATINS  TO  SELECTION 
OF  QUANTITATIVE  RELIABILITY  REQUIREMENTS 


Air  Force 
Document 

Level 
to  Which 
Assigned 

Statement  of  Responsibility 

Par. 

No. 

AFR  80-5 

4  June  1962 

Hq,  USAF 

(l)  Insure  that  specific  operational 
requirements  and  system  documentation 
Include  numerical  reliability  require¬ 
ments  and  adequate  provisions  far 
reliability  programs. 

5-b 

AFSC 

(l)  Establish  objectives  for  system 
analysis  and  quantitative  reliability 
requirements  for  use  in  the  conceptual 
phase. 

6.b 

(2)  Incorporate  quantitative  relia¬ 
bility  requirements  and  provisions  for 
demonstrating  reliability  in  all  speci¬ 
fications,  exhibits,  product  descrip¬ 
tions,  work  statements  and  contractual 
clauses  to  be  referred  to  or  included 
in  contracts  for  systems  and  associated 
material. 

6.  c 

AFSCR  80-1 

14  Dec.  1962 

AFSC 

Divisions 
and  Centers 

(l)  Develop  and  incorporate  quantitative 
reliability  requirements  in  PSPP's, 

SPP' s,  DP's*,  and  other  programming 
documents  that  may  be  applied  to  indivi¬ 
dual  systems. 

6. a 

System 

Project 

Offices 

(SPO’s) 

(1)  Establish  with  the  collaboration 
of  the  operating  command  and  technical 
assistance  from  the  development  division 
realistic  quantitative  reliability  re¬ 
quirements  for  the  system. 

7.b 

afbsor  aoT" 
28  Dec.  1962 

SE/TD 

Contractors 

(1)  Establish  numerical  reliability 
requirements  for  each  contractor. 

6.c 

*PSPP  ■  Proposed  System  Package  Plan 

SPP  ■  System  Package  Program 

IV  ■  Development  Plan 

2. 1.1. 4  Implementation  of  Quantitative 


Requirements  Selection 


The  following  steps  represent  the  basic  procedure 
for  selecting  a  quantitative  reliability  requirement  for 
a  defined  system: 

(1)  Determine,  to  the  extent  possible,  the  mission 
of  the  system. 

(2)  Determine  the  need  for  establishing  a  quantita¬ 
tive  requirement. 

(3)  Determine  the  form  in  which  the  quantitative 
requirement  can  best  be  stated  (e.g.,  mean  time 
between  failures,  probability  of  survival,  etc.). 

(4)  Determine  the  number  and  nature  of  the  require¬ 
ments  to  be  stated  (e.g.,  possible  utilization 
of  a  higher  reliability  goal  in  addition  to 
minimum  acceptable  and  intermediate  values). 

(5)  Perform  trade-off  analyses  between  reliability 
and  other  parameters  (e.g.,  system  cost,  weight 
and  space  limitations,  etc.). 

(6)  Determine  -the  requirements  for  reliability 
demonstration. 

(7)  Determine  statistical  confidence  levels  to  be 
associated  with  the  quantitative  requirements. 

(8)  Perform  an  estimate  of  attainable  reliability. 

(9)  Prepare  a  formal  statement  of  the  quantitative 
reliability  requirement  for  the  system. 

The  above  listing  generally  represents  the  sequence  in 
which  the  procedural  steps  should  occur,  but  deviations 
to  this  order  will  not  necessarily  prevent  successful 
accomplishment  of  the  task. 

The  separate  steps  in  selecting  a  quantitative  re¬ 
quirement  are  discussed  in  detail  in  following  sections. 
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2. 1.1. 4.1  Description  of  System  Mission 


Numerical  reliability  Is  associated  with  the  concept 
of  system  mission  (see  Appendix  P) .  Hence,  in  determining 
a  quantitative  reliability  requirement  for  a  system,  the 
system's  mission  must  first  be  defined  arid  understood. 
Although  the  full  details  of  the  mission  may  not  be  estab¬ 
lished  until  a  later  time,  some  information  should  never¬ 
theless  be  available  at  the  time  when  quantitative  relia¬ 
bility  is  initially  considered. 

The  specific  mission  characteristics  which  should  be 
defined  in  order  to  develop  a  realistic  reliability  re¬ 
quirement  include: 

(1)  The  function  to  be  performed  by  the  system 
(or  its  subsystems  or  major  components). 

(2)  The  intended  operating  conditions  and/or 
environment . 

(3)  The  duration  of  the  mission. 

(4)  The  feasibility  of  repair  of  the  system  before, 
during,  and  after  the  mission. 

(5)  The  intended  useful  life  of  the  system,  either 
in  time  or  number  of  missions  to  be  performed. 

Regardless  of  the  time  at  which  these  characteristics  are 
considered,  the  information  available  on  them  can  be  ex¬ 
pected  to  vary  in  degree  of  aoeuraoy  and  completeness. 
However,  they  should  be  defined  to  the  extent  possible 
with  such  information. 

Various  formal  and  Informal  documents  will  exist 
prior  to  any  contract  award  which  can  provide  assistance 
in  describing  the  system's  mission.  Typical  types  of 
documents  (the  document  titles  may  vary,  from  program  to 
program)  which  will  normally  be  avt liable  include: 

(1)  Speolflo  Operational  Requirement  (SOR)  — 

This  document  states  a  need  for  a  capability, 
outlines  a  system  or  major  component  for 
achieving  it,  and  gives  reasons  for  the 
requlresMnt. 


© 
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(2)  Advanced  Development  Objective  (ADO)  — 

This  document  outlines  an  experimental 
system  or  major  component  which  is  not 
yet  assured  as  to  military  usefulness, 
technical  feasibility,  and  financial 
acceptability . 

(3)  Proposed  System  Package  Plan  (PSPP)  — 

This  document  identifies  the  responsi¬ 
bilities,  tasks,  resources,  and  time 
phasing  of  the  major  actions  of  each 
Air  Force  organization  participating 
in  a  system  program. 

In  addition,  the  conceptual  phase  of  a  program  will  be 
characterized  by  a  series  of  correspondence,  reports, 
or  descriptive  statements  which  will  contribute  to  over¬ 
all  knowledge  of  the  mission  requirements. 


2. 1.1. 4. 2  Need  for  Establishing 

Quantitative  Reliability  Requirement 


The  need  for  establishing  quantitative  reliability 
requirements  is  generally  applicable  to  Air  Foroe  re¬ 
search  and  development  for  systems,  subsystems,  and 
equipment  (see  Table  2-1).  There  are  few  situations 
when  a  deoislon  should  be  made  to  negate  formulation  of 
quantitative  requirements.  In  the  event,  however,  that 
such  a  decision  is  deemed  in  the  best  interests  of  the 
Air  Force,  it  must  be  fully  Justified,  and  the  ultimate 
decision  will  warrant  aotlon  by  high  levels  of  authority. 


2. 1.1. 4. 3  Form  of  Expressing  Quantitative 
Requirements 


There  are  several  forms  in  which  numerical  reliability 
requirements  can  be  expressed.  Table  2-3  lists  and  defines 
the  more  common  forms,  and  gives  oertaln  guidelines  to 
assist  in  determining  whloh  form  of  expression  is  appro¬ 
priate  to  a  particular  selection.  There  is  no  general 
restriction  in  Air  Faroe  polioy  which  dictates  the  speolflo 
manner  in  which  the  quantitative  reliability  requirement 
for  a  system  is  to  be  expressed  (although  BSDR  70-16, 
Contractual  Reliability  Requirements,  dated  26  Deoeaber  1962, 
nrsasiibss  tbs  use  of  "probability  of  mission  suooess"  and/or 
"■wan  tins  between  failures"). 
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TABLE  2-3 


FORMS  07  EXPRESSION  FOR  NUMERICAL  RELIABILITY 
AND  GUIDELINES  FOR  SELECTING  FORMS 
TO  BE  USED  IN  STATING  QUANTITATIVE  RELIABILITY  REQUIREMENTS 


Form  of  Expression 

Definition 

Mean  time  to  failure 

The  average  or  mean  life  of  an  Irreparable 
item  (expressed  in  hours  or  cycles) 

Mean  time  before  failure 
(or  between  failures) 

The  limit  of  the  ratio  of  operating  time 
to  the  number  of  failures,  as  the  number 
of  failures  approaches  infinity  (expressed 
in  hours  or  cycles)  — , 

Reliability  of  equipment 

The  probability  that  an  item  will  satis¬ 
factorily  perform  a  given  function  for  a 
definite  period  of  time  under  specified 
conditions  (expressed  as  a  percentage  or. 
a  decimal  fraction  of  l) 

Probability  of  mission 
success 

The  probability  that  a  given  item  will 
satisfactorily  perform  a  stated  mission 
(expressed  as  a  percentage  or  a  decimal 
fraction  of  l) 

Failure  rate 

The  number  of  failures  per  unit  of  operat¬ 
ing  time  (usually  expressed  as  the  per 
cent  of  item  failures  during  a  1000-hour 
interval ) 

Guidelines  for  Selecting  Form  of  Expression  to  be  Used 

(1)  A  statement  of  numerical  reliability  in  terms  of  "mean 
time"  implies  a  known  or  assumed  distribution  of  failure 
times.  For  example.  In  electronic  equipment,  failures 
are  usually  assumed  to  occur  at  a  random  rate. 

(2)  A  statement  of  "mean  time,"  when  accompanied  by  informa¬ 
tion  concerning  the  distribution  of  failure  times,  can 
be  easily  converted  to  a  probability  statement. 

(3)  "Mean  time  to  failure"  is  generally  used  only  in  con¬ 
junction  with  an  irreparable  item. 

(4)  "Probability  of  mission  success"  is  generally  used  for 
one-shot  Items  for  which  the  mission  is  known  and  success 
can  be  defined.  Its  use  Is  not  restricted  to  such  situa¬ 
tions,  however. 

(5)  "Failure  rate"  is  most  often  associated  with  simple 
levels  of  consideration  (e.g.,  parts). 

(6)  "Mean  time  before  failure"  is  the  reciprocal  of  "failure 
rate"  when  the  failure  rate  is  constant. 

(7)  "Probability"  can  be  associated  with  either  a  given  time 
(or  number  of  oyoles)  or  a  defined  mission,  whichever 
represents  the  most  adequate  description  of  system  re¬ 
quirements  . 
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2. 1.1. 4. 4  Number  and  Nature  of  Quantitative 
Requirements 


The  ultimate  objective  of  this  task  Is  to  determine 
whether  primary  and/or  sole  emphasis  will  be  placed  on 
the  formulation  of: 

(1)  A  single  numerical  reliability  requirement 
(usually  called  "minimum  acceptable  value" ), 
or 

(2)  A  series  of  discrete  graduated  levels  of 
numerical  reliability  as  based  on  anticipated 
advancement  of  the  state  of  the  art  or  planned 
progress  (usually  oalled  the  "Idealised  relia¬ 
bility  growth  curve"  and  depicted  as  shown  in 
Figure  2-1). 

The  later  method  (2)  of  establishing  numerical  requirements 
Is  stipulated  within  AFR  80-5  and  AFSCR  80-1  (see  Table  2-1). 
When  the  growth  ourve  Is  used  to  describe  numerical  require¬ 
ments,  the  point  or  points  on  the  ourve  that  are  to  be 
accompanied  by  reliability  demonstration  should  be  Identified. 


2. 1.1. 4. 5  Relationship  of  Reliability 

to  Other  System  Parameters  ("Trade-Offs") 


The  development  of  quantitative  reliability  require¬ 
ments  for  a  system  requires  consideration  of  restricting 
relationships  between  reliability  and  other  system  para¬ 
meters.  The  prlnolpal  system  parameters  which  affeet 
reliability  are  shown  In  Figure  2-2  and  dlsousaed  In  fol¬ 
lowing  seotlons. 


2. 1.1. 4. 5.1  Development  Time 


Since  the  relationship  of  development  time  to  reliar- 
bility  is  difficult  to  assess  quantitatively,  the  assump¬ 
tion  is  often  made  that  development  time  does  not  affect 
reliability.  However,  there  is  qualitative  evidence  that 
drastic  shortening  of  development  schedules  adversely 
affects  system  reliability. 

Figure  2-3  shows  that  a  gradual  increase  in  relia¬ 
bility  occurs  as  a  program  progresses  from  the  feasibility 
stage  to  the  production  stage.  Ibis  reliability  growth 
results  from: 

(1)  Redesign  of  Immature  or  weak  components. 

(2)  Elimination  of  part  misapplications. 

(3)  Improvement  of  part  and  system  quality. 

(4)  Elimination  of  Interfaces,  cabling,  and  other 
system  type  problems  which  reduce  reliability. 

(5)  Elimination  or  Improvement  of  poor  reliability 
items  discovered  through  demonstration  and 
environmental  tests. 

(6)  Improvement  of  manufacturing  and  inspection 
procedures. 

Shortening  or  compression  of  development  schedules  does 
not  allow  for  the  natural  progression  of  the  above  factors 
and  is  therefore  likely  to  Involve  compromises  with  respect 
to  reliability. 

If  schedules  must  be  established  which  do  not  permit 
completLonof  desired  reliability  activity,  measures  must 
be  taken  to  compensate  for  the  decrease  in  reliability. 
These  measures  may  Include  more  careful  maintenance,  pro¬ 
tection  from  environment,  special  instructions  to  the  user, 
or  increase  in  production  quantities. 


2. 1.1. 4. 5. 2  Complexity 


As  the  complexity  of  an  equipment  increases,  its 
reliability  deoreasth;  further,  the  reliability  decrease 
ia  more  rapid  then  tne  complexity  increase  (see  Pigure  2-10 
in  S.l.1.4.8). 


2. 1.1. 4. 5. 3  Availability 

Availability  is  the  probability  that  a  system  (subsystem,  equipment, 
device,  etc.)  will  be  operable  at  a  desired  instant  of  time  when  used  under 
specified  conditions.  This  probability  statement  is  concerned  with  an 
instant  or  point  in  time  rather  than  a  duration  or  a  given  interval;  this 
is  the  significant  difference  between  availability  and  reliability.  Also, 
the  term  "availability"  is  concerned  vith  vhether  the  system  is  available 
to  operate  at  a  given  instant  regardless  of  vhether  it  is  actually  in 
operation.  Thus,  if  the  system  is  in  a  good  state  of  repair,  it  makes 
no  difference  vhether  it  is  "turned  on"  or  "turned  off,"  provided  that 
sufficient  "warm  up”  time  can  be  planned  prior  to  initiating  a  mission. 

Availability  can  be  estimated  from  the  ratio 
A  =  e/  ( 0  +  ♦) , 
where  6  =  system  MTBF. 

and  p  *  system  mean  "down  time." 

This  is  an  estimator  rather  than  a  definition.  It  is  a  rigorous  definition 
only  if  the  exact  values  of  6  and  can  be  found. 


2. 1.1.4. 5. 4  Maintainability 

Maintainability  is  the  probability  that,  when  maintenance  action 
is  initiated  under  stated  conditions,  a  failed  system  will  be  restored 
to  operable  condition  within  a  specified  total  down  time. 

Maintainability,  by  this  definition,  is  synonymous  with  the  usual 
meaning  of  the  word  "repairability"  or  reparability."  The  time  duration 
of  interest  in  this  definition  must,  itself,  be  precisely  described.  In 
same  usages  of  the  term  "maintainability,"  only  that  time  duration  which  a 
repair  operation  is  actively  being  carried  out  1b  considered;  and  admini¬ 
strative  time  and  logistic  time  are  omitted.  However,  inclusion  or  emission 
of  these  additional  periods  of  time  may  be  left  to  the  desires  or  needs  of 
the  moment,  provided  the  conditions  are  clearly  stated. 

Estimation  of  maintainability  is  essentially  identical  to  estimation 
of  reliability.  To  ecr-ess  maintainability  of  an  existing  system,  one 
rav/'t  observe  the  length  of  time  each  maintenance  action  requires;  from 
such  data  a  maintainability  function  may  be  generated  by  non-parsmetric 
means,  or  parameters  may  be  estimated  if  a  distribution  has  been  assumed. 

To  predict  maintainability,  one  must  define  certain  ground  rules  and 
assumptions  upon  which  to  base  derivation  of  a  mathematical  model  of  the 
maintainability  function  of  the  system.  Into  this  model  may  be  Inserted 
data  derived  from  past  experience  with  maintenance  of  similar  systems; 
this  constitutes  a  prediction  of  maintainability . 
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The  relationship  between  reliability  and  maintainability  must  be 
discussed  in  terms  of  availability  (see  Figure  2-4).  The  graph  shows 
that,  for  a  given  value  of  availability,  a  decrease  in  reliability 
(expressed  as  a  decrease  in  mean-time-between-fallures)  can  be  traded- 
off  by  a  corresponding  Increase  in  maintainability  (expressed  as  a 
decrease  in  mean-time-to-repair).  Conversely,  if  reliability  can  be 
increased,  requirements  for  maintainability  can  be  relaxed  while 
maintaining  the  same  availability. 

Figure  2-5  illustrates  the  trade-off  discussed  in  the  preceding 
paragraph.  If  point  0  in  the  figure  represents  the  status  of  a  design 
with  respect  to  reliability  and  maintainability,  an  improvement  in 
reliability  can  have  one  of  several  effects:  maintainability  can  be 
improved  (a),  kept  constant  (b),  or  reduced  (c),  depending  on  how 
repair  times  are  affected  by  the  design  change.  Thus,  reliability 
changes  in  the  conceptual  design  phase  must  be  carefully  evaluated  as 
to  their  effect  on  maintainability  and  availability. 

MIIf-M-26512C  (USAF),  "Maintainability  Program  Requirements  for 
Aerospace  Systems  and  Equipment”  establishes  maintainability  program 
requirements  and  policies.  Included  is  a  detailed  procedure  for  prediction, 
testing  and  demonstration  of  availability  and  maintainability. 
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FIGURE  2-4 

RELATIONSHIP  BETWEEN  RELIABILITY  (EXPRESSED  IN  TERMS  OF  MTBF), 
MAINTAINABILITY  (EXPRESSED  IN  TERMS  OF  MTTR),  AND  AVAILABILITY 


100 


Ftssn-Ti*s-Between-F«ilurt»  (Hours) 
2.1-16 


FIGURE  2-5 


RELIABILITyAAINTAINABILIT*  TRADE-OFF 


2. 1.1. 4. 5. 5  Coat 


Figure  2-6  shows  the  characteristic  relationship 
between  system  cost  and  reliability:  as  system 
reliability  Increases,  operational  costs  decrease  but 
development  costs  Increase.  The  total  cost  of  a  system 
program  Is  the  summation  of  development  and  operational 
costs,  and  the  most  efficient  program  cost/rellablllty 
relationship  Is  represented  by  the  lowest  point  on  the 
cost  summation  curve  In  Figure  2-6.  This  trade-off 
effectively  Illustrates  that  when  reliability  Is  related 
to  cost,  both  procurement  and  maintenance  costs  must  be 
considered . 

When  costs  are  strictly  limited  on  a  program  and 
some  reliability  activity  must  be  curtailed,  such  pre¬ 
cautions  as  special  Instructions  to  the  user  and  more 
frequent  maintenance  activity  should  be  followed  to 
compensate  for  loss  of  reliability. 


2*1. 1.4. 5. 6  Weight  and  Size 


Weight  and  size  are  factors  whlqh  must  often  be 
traded-off  with  reliability,  particularly  in  airborne 
and  satellite  applications. 

Size  and  weight  reductions  can  be  made  through  the 
use  of  miniaturized  and  low-power  components,  but  these 
changes  are  not  always  without  adverse  effects  on  relia¬ 
bility.  Indlscrimlnant  reductions  in  size  and  weight 
can  cause  short  olrouits,  incorrect,  wiring,  overheating, 
•  overstressing,  and  Increased  maintenance. 
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On  the  other  hand,  when  reliability  is  a  paramount 
consideration,  as  in  safety  applications,  it  can  be  im¬ 
proved  through  weight  and  size  increases  in  the  form  of 
redundancy.  Figure  2*7  shows  that  if  a  unit  with  a 
reliability  of  0.6  is  applied  in  redundant  configurations 
of  2,  3>  4 ,  ---  units,  the  reliability  of  the  over-all 
configuration  is  rapidly  increased.  Assuming  that  no 
weight  is  required  for  switching  the  redundant  units,  the 
horizontal  scale  in  the  figure  can  be  read  in  terms  of 
relative  weight  or  size . 


2. 1.1. 4. 5.  7  Environment 


Although  environment  may  not  be  strictly  considered 
as  a  system  parameter,  it  nevertheless  has  an  Important 
effect  on  reliability.  In  general,  reliability  decreases 
as  the  severity  of  environment  is  increased  (see  Figure  2-8). 
Thus,  if  there  is  no  way  to  influence  the  applied  environment, 
the  choice  of  parts  and  components  should  be  made  on  the 
basis  of  maximum  performance  in  the  most  severe  environment 
expected  to  be  encountered. 


Demonstration 


It  is  impractical  to  consider  quantitative  reliability 
as  a  program  requirement  unless  there  1b  a  provision  for 
reliability  demonstration.  A  reliability  level  which  is 
not  to  be  realized  through  demonstration  must  be  considered 
a  goal,  rather  than  a  requirement  which  is  binding  on  a 
contractor. 

Full  details  for  reliability  demonstration  will  usually 
not  be  developed  until  some  time  after  a  contract  is  awarded. 
However,  during  the  formulation  of  reliability  requirements, 
it  is  desirable  to  set  forth  as  much  information  as  possible 
concerning : 

(1)  The  Intended  time  of  demonstration,  or  time 

at  which  demonstration  will  first  become  feasible. 

(2)  The  extent  of  demonstration  —  whether  it  is  to 
be  a  one-time  occurrence  or  a  planned  series  of 
activities.  If  the  demonstration  is  planned  as 

a  series  of  progressive  steps,  these  steps  should 
ooinclde  with  major  program  milestones,  such  as 
a  preproduction  demonstration  or  a  service- 
readiness  demonstration. 
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FIGURE  2-7 


RELATIONSHIP  BETWEEN  RELIABILITY 
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2. 1.1.  k. 7  Confidence  Levels  for  Quantitative  Requirements 

A  quantitative  statement  of  required  reliability  can  be 
formulated  for  a  system  without  expressing  statistical  confidence 
levels.  However,  as  discussed  in  2. 1.1. 4. 6,  a  quantitative  relia¬ 
bility  requirement  is  intended  to  be  demonstrated,  and  for  the 
process  of  demonstration,  it  is  necessary  to  indicate  a  required 
confidence  level  (i.e.,  degree  of  assurance,  expressed  as  a 
percentage,  that  the  reliability  value  measured  in  the  demon¬ 
stration  test  is  the  actual  reliability  of  the  system). 

Figure  2-9  represents  an  example  of  graphically  illustrat¬ 
ing  the  quantitative  relationship  between  statistical  confidence 
level,  reliability  requirement,  and  parameters  of  reliability 
demonstration  such  as  number  of  failures  encountered  and  required 
operating  time.  This  example  is  restricted  to  the  assumption 
that  only  one  failure  is  allowed  during  demonstration.  In  this 
figure,  reliability  is  expressed  as  a  probability  of  mission 
success,  with  the  number  of  equivalent  missions  required  for  demon¬ 
stration  plotted  as  a  function  of  required  mission  reliability. 
Plots  cure  shown  for  several  assumed  levels  of  desired  confidence. 

Figure  2-9  is  not,  of  course,  of  general  application. and 
is  only  to  be  used  if  all  conditions  are  met, including  the 
assumption  that  only  one  failure  is  allowed.  It  has  been 
introduced  at  this  point  solely  to  show  that  in  establishing  a 
quantitative  reliability  requirement,  there  must  be  an  accom¬ 
panying  understanding  of  the  implications  with  respect  to  relia¬ 
bility  demonstration  and  statistical  Confidence. 

2. 1.1. ' 4. 8  Estimates  of  Attainable  Reliability 

The  process  of  estimating  or  "predicting"  the  relia¬ 
bility  of  a  new  system  is  essential  to  the  formulation  of  a 
quantitative  reliability  requirement  for  the  system.  In  some 
instances,  the  reliability  prediction  technique  serves  as 
the  sole  basis  for  the  initial  statement  of  required  reliability. 

In  other  Instances,  where  a  quantitative  requirement  has  been 
determined  by  some  other  means  (such  as  a  definite  tactical  need 
or  an  arbitrary  decision  as  to  required  operational  capability), 
the  prediction  technique  can  be  utilized  for  comparative  analysis. 

During  the  conceptual  phase  of  a  program,  comparatively 
little  data  will  be  available  to  serve  as  the  basis 
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for  estimating  attainable  reliability.  The  standard 
approach  to  predicting  reliability  of  a  new  system  is, 
therefore,  to  investigate  past  performance  of  identical 
or  similar  systems.  The  following  are  three  examples 
of  techhiques  employing  this  approach: 

(1)  Figure  2-10,  taken  from  MIL-STD  756A,  Relia¬ 
bility  Prediction  Procedures  for  Aircraft, 
Missiles,  Satellites,  and  Electronic  Equipment , 
dated  'Ip  May  1963*  This  chart  is  intended  to 
provide  a  method  of  prediction  based  on  general 
considerations  of  environment  and  equipment 
complexity.  It  was  derived  through  interpreta¬ 
tion  of  observed  data  for  systems  operating 
(prior  to  i960)  in  ground,  shipborne,  and  air¬ 
borne  environments . 

(2)  Figure  2-11,  taken  from  Aerospace  Corporation 
Report  No.  ATM-63 ( 3303 )-l.  Reliability  Prac^ 
tlces  and  Problems  for  Spacecraft  and  Missile 
Systems,  dated  1  October  l9fe2.  The  information 
plotted  in  this  curve  was  derived  from  an 
analysis  of  mission  success  associated  with  the 
launching  of  U.  S.  space  vehicles. 

(3)  A  presentation  of  observed  reliability  for  cer¬ 
tain  Air  Force  ground  electronic  equipments  as 
a  function  of  equipment  complexity  (in  number 
of  system  parts),  in  an  article  entitled,  "The 
Measurement  and  Specification  Of  Product  Abil¬ 
ities,"  by  F.  A.  Cafaro  and  H.  D.  Voegtlen,  in 
Industrial  Quality  Control,  March  1962. 

These  examples  are  not  cited  to  suggest  prediction 
tools  which  have  general  application.  Rather,  they  are 
mentioned  to  illustrate  that  techniques  for  analyzing  and 
summarizing  data  from  past  performance  have  been  and  are 
continuing  to  be  developed.  Hence,  when  formulating  a 
quantitative  requirement  for  a  specific  new  program,  it 
is  advisable  to  exploit  available  sources  in  determlng 
the  applicability  of  observed  past  performance. 

Any  estimate  or  prediction  of  system  reliability 
should,  of  course,  be  related  to  the  point  in  time  at 
whloh  attainment  is  implied.  For  example,  if  the  esti¬ 
mate  is  based  upon  anticipated  advancement  of  the  state- 
of-the-art,  the  expression  of  the  estimate  should  be  so 
qualified . 


FIGURE  2-10 

FAILURE  RATE  AND  MTBF  VERSUS  FUNCTIONAL  COMPLEXITY 
FOR  ELECTRONIC  EQUIPMENT* 


Notea :  (l)  Active  element  group  (AEGVi  one  electron  tube  or  tran¬ 

sistor  and  associated  circuitry  (or  ten  computer 
diodes  and  associated  circuitry  in  digital  com¬ 
puters  ) 

(2)  Reliability  estimate  obtained  from  this  chart 
represents  band  of  possible  outcomes.  Upper 
limit  of  band  is  obtainable  with  good  relia¬ 
bility  and  design  effort. 

•Figure  taken  from  MIL-STD  7 56 A. 
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Mean  Time  Between  Failures  (MTBF)  in  Hours 


TOTAL  MISSIONS  VERSUS  SUCCESSFUL  MISSIONS 
U.  S.  SPACE  VEHICLES  (INCLUDING  BOOSTERS)* 


2.1. 1.4.9  Statement  of  Quantitative 
Requirement 

The  culmination  of  the  efforts  described  in  2. 1.1. 4.1 
through  2. 1.1. 4. 8  is  a  formal  statement  of  quantitative 
reliability  requirements  for  a  particular  program.  In  set¬ 
ting  forth  this  statement,  the  attempt  should  be  made  to 
express  the  requirement  in  a  manner  suitable  for  inclusion 
in  a  contract. 

The  actual  numerical  statement  of  the  reliability  re¬ 
quirement  will  most  likely  be  brief,  as  typified  by  the 
following  examples: 

Sample  Statement  (l) 

The  minimum  acceptable  reliability  requirement  for 
the  system  is  90$  probability  of  success,  demonstrated 
to  a  90  per  cent  confidence  level  for  the  mission 
profile  as  defined  in  (cite  reference). 

Sample  Statement  (2) 

The  minimum  acceptable  reliability  for  the  system 
is  as  follows  (constant  failure  rate  assumed): 


Configuration 

MTBF  (hr.) 

Confidence  Level  ($) 

X 

500 

90 

Y 

750 

90 

Z 

1000 

80 

^Demonstration  Requirement 
Sample  Statement  (3) 

The  minimum  acceptable  reliability  requirement  for 
the  system  is  0.95  for  a  typical  mission  with  a 
duration  of  4  hours,  demonstrated  to  a  90  per  cent 
confidence  level.  The  typical  mission  shall  be  as 
defined  in  (cite  reference). 

Sample  Statement  (4) 

The  failure  rate  for  the  part  shall  be  no  greater 
than  0.01$  per  1000  hours  of  operation,  demonstrated 
to  a  60  per  cent  confidence  leve-. 

The  brief  numerical  statement  must,  however,  be  aug¬ 
mented  with  considerable  amplifying  information  such  as 
assumptions,  qualifications,  definitions,  and  stipulations. 
This  information  could  include  any  or  all  of  the  following 

items: 
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(1)  Description  of  mission  profile. 

(2)  Description  of  environmental  conditions. 

(3)  Statement  of  type  of  failure  distribution. 

(4)  Statement  of  program  phase  in  which  reliability 
demonstration  is  Intended. 

(5)  Comparison  of  reliability  requirement  to  known 
or  anticipated  state-of-the-art. 

(6)  Definition  of  mission  success  or  failure. 

(7)  Description  of  equipment  to  which  reliability 
requirement  applies. 

(8)  Amplification  of  intended  contractual  significance 
of  reliability  requirements 

(9)  In  incentive  contracts  having  reliability  penalties  and 
premiums,  include  a  statement  of  minimum  acceptable 
statistical  confidence  level  for  demonstration. 


2. 1.1. 5  Problem  and  Special  Considerations 

Relating  to  Quantitative  Requirements  Selection 

2.1. 1.5.1  Criticalness  of  Reliability 

It  is  possible  to  identify  seme  situations  in  which,  based 
on  subjective  evaluation,  reliability  is  less  critical  than  are 
other  factors.  There  are  certain  systems,  for  example,  in  which 
availability  or  maintainability  are  considered  to  take  precedence 
over  reliability.  For  such  systems,  management  decisions  may  be 
appropriate  to  relax  quantitative  reliability  requirements  in  order 
to  improve  or  concentrate  emphasis  on  the  other  system  effective¬ 
ness  and  associated  parameters). 

The  criticalness  of  reliability  1b,  of  course,  related  to 
the  existing  state-of-the-art  for  reliability  achievement.  But 
there  are  factors  other  than  state-of-the-art  which  dictate 
required  numerical  reliability.  For  example,  the  criticalness 
of  the  reliability  requirement  may  be  increased  if  the  tactical 
situation  demands  reliability  in  excess  of  that  attainable 
through  normal  or  routine  practice.  This  problem  occurred  in  the 
Minuteman  program,  and  the  necessary  Increase  in  capability  was 
taken  into  consideration  in  planning  the  program  effort. 
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2. 1.1. 5. 2  State  of  Program  Maturity 


In  2. 1.1. 4,  it  was  assumed  that,  in  general,  the 
process  of  selecting  reliability  requirements  occurs 
early  in  the  conceptual  phase  of  a  particular  program. 

This  assumption  was  based  on  the  fact  that  a  quantitative 
requirement  must  exist  at  the  time  a  program  is  opened 
to  bids.  Often,  however,  the  need  arises  to  refine, 
amplify,  or  re- state  the  requirement  at  a  later  stage  of 
the  program.  The  guidance  principles  and  procedures  set 
forth  In  2. 1.1. 4  are  applicable  to  requirements  initiation 
or  review  at  any  period  during  the  life  of  a  program. 


2. 1.1. 5. 3  Interpretation  of  Reliability 
Requirements 


Different  interpretations  placed  on  the  significance 
of  a  numerical  reliability  statement  by  the  contractor 
and  the  customer  are  a  common  source  of  difficulty  in  con¬ 
tracted  programs .  Frequently,  a  contractor  assumes  that 
reliability  values  are  intended  to  represent  desired 
achievements  or  goals,  while  the  customer  has  Intended 
that  they  represent  unequivocal  obligations.  To  avoid 
such  difficulty,  the  customer  must: 

(1)  Emphasize  the  obligatory  Intent  in  expressing 
reliability  requirements  in  contracts. 

(2)  Stipulate  reliability  requirements  that  are, 
in  fact,  realistic. 

(3)  Support  the  requirements  by  including  a  re¬ 
quirement  for  a  feasible  reliability  demon¬ 
stration  program. 


2. 1.1. 5.4  Inclusion  of  Quantitative  Requirements 
in  Incentive  Clauses 


The  selection  of  quantitative  reliability  requirements 
is  complicated  when  they  are  intended  for  inclusion  in  in¬ 
centive  contracts  —  and  particularly  when  reliability 
forms  the  basis  for  incentive.  In  such  case,  it  is  necessary 
to  determine  a  "standard"  requirement  which  can  be  related  to 
a  "standard"  fee.  In  addition,  the  lnorsase  or  deorease  in 
fee  as  a  function  of  the  increase  or  deorease  in  achieved 
reliability  must  be  established. 
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Considerable  effort  has  been  advanced  to  develop 
reliability  incentives,  and,  based  on  current  policy 
statements  in  APR  80-5,  it  is  apparent  that  such  in¬ 
centives  will  be  extensively  applied  in  future  pro¬ 
curements  . 


2. 1.1. 6  Additional  References  on  Quantitative 
Requirements  Selection 


Specification  and  Assurance  of  Large  MTBF*s 
Typical  o t  Spacecraft  Electronic  Equipments, 

CVc.  Petersen,  Military  Systems  Design,  L 
April  1963,  pp.  27-33. 

Reliability  Practices  and  Problems  for  Space¬ 
craft  and  Missile  Systems,  fr.  77  Klein,  Aerospace 
Corporation  Report  No.  A'tM-63(3303)-l, 

1  October  1962. 

Quantitative  Reliability  Requirements,  Letter 
from  B.  AT  Schr lever.  General,  USAF,  Commander 
APSC,  25  January  1962. 
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$  2.1.2  Selection  of  Qualitative 

Reliability  Requirements 


2. 1.2.1  Definition 


The  current  approach  to  military  procurement  Is  to 
combine  quantitative  requirements  (which  stipulate  measur¬ 
able  characteristics)  with  requirements  which  specify  in 
detail  the  elements  of  work  and  level  of  effort  to  be 
implemented  in  a  program.  The  specification  of  such  quali 
tative  requirements  for  a  reliability  program  is  not  in¬ 
tended  to  limit  the  initiative  of  a  contractor  or  relieve 
him  of  his  responsibility  for  achieving  a  particular 
quantitative  reliability  requirement.  Rather,  it  is  in¬ 
tended  to  support  the  contractor's  effort  by  providing 
adequate  direction  and  enabling. a  common  understanding 
of  the  scope,  objective,  and  progressive  achievement  of 
the  program. 


Selection  of  qualitative  reliability  requirements, 
as  discussed  in  2.1.2,  comprises  that  initial  activity 
which  is  directed  toward  outlining  ,  choosing,  or  develop¬ 
ing  the  elements  which  will  govern  the  scope  of  work  and 
level  of  effort  of  a  planned  reliability  program.  This 
activity  commences  during  the  conceptual  phase  of  the 
program  and  continues  until  adequate  information  exists 
to  permit  preparation  of  a  statement  of  work  or  similar 
contractual  instrument. 


2. 1.2.2  Air  Force  Policy 


Air  Force  policy  which  relates  to  selection  of  quali¬ 
tative  reliability  requirements  is  summarized  in  -Table  2-4 


2. 1.2. 3  Responsibilities  for  Qualitative 
Requirements  Selection 


The  responsibilities  of  System  Program  Offices  (and 
thus  SPO  Reliability  Coordinators)  and  other  associated 
levels  of  authority  relating  to  selection  of  qualitative 
requirements  are  summarized  in  Table  2-3. 
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TABLE  2-4 

'  SUMMARY  07  AIR  FORCE  POLICY 
RELATING  TO  SEIZCTION 
OF  QUALITATIVE  RELIABILITY  REQUIREMENTS 


Air  Force 
Document 

Statement  of  Policy 

Par. 

No. 

AFR  80-5 

4  June  1962 

(l)  Inherent  reliability  le  established  by  the 
basic  design  and  can  be  Improved  only  by  design 
changes.  The  feasible  time  to  Implement  relia¬ 
bility  Is  during  design  and  early  development 
and  testing. 

3.b 

(2)  Reliability  will  be  stressed  during  early 
system  studies,  source  selection,  design,  devel¬ 
opment,  and  production.  It  Is  of  major  conse¬ 
quence  during  the  operational  phase  of  the 
system,  but  Its  adequacy  In  that  phase  depends 
upon  the  emphasis  received  during  the  conceptual 
and  acquisition  phases . 

*.M1) 

(3)  During  the  conceptual  and  acquisition  phases, 
systems  will  be  analyzed,  and  a  reliability 
established  for  each. 

4.a.(3) 

(4)  System  contracts  will  Include  a  requirement 
for  a  comprehensive  contractor  reliability 
program. 

4.c 

AFSCR  80-1 

14  Dec.  1962 

(l)  Proposed  system  package  plans  (PSPP's)  sys¬ 
tem  package  programs  (SPP's),  and  development 
plans  (DP's)  will  contain  a  narrative  statement 
delineating  the  desired  reliability  character¬ 
istics.  However,  comprehensive  reliability 
programs  for  feasibility  studies,  exploratory 
development,  and  advanced  development  categories 
are  not  desired.  Due  consideration  shall  be 
given  to  reliability  In  the  early  planning  and 
feasibility  study  stages,  and  comprehensive 
reliability  programs  are  expected  for  operational 
development  projects. 

4.b 

(2)  Contracts  for  aerospace  systems,  major  sub¬ 
systems,  and  equipment  will  include  a  requirement 
for  a  comprehensive  and  organized  contractor 
reliability  program  that  extends  through  sub¬ 
contractor  and  vendor  levels. 

4.e 

(3)  Specific  activities  may  be  Identified  as 
reliability  effort  In  order  to  provide  a  manage- 

effort . 

4.f 

Ansnt  80-5 
28  Dec.  1968 

(l)  Specific  reliability  prograsia  will  be  estab¬ 
lished  for  all  balllstios  systems  or.  portions 
thereof  having  separate  Air  Force  contracts  for 
their  development/procurement . 

5 

(2)  MXL-R-27542  will  be  Incorporated  into  future 
Air  Force  ballistic  missile  contracts  and  into 
sxlstlnt  contracts  as  appropriate. 

5-c 
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TABLE  2-5 

SUMMARY  OF  RESPONSIBILITIES 
RELATING  TO  SELECTION 
OF  QUALITATIVE  RELIABILITY  REQUIREMENTS 


Air  Force 
Document 

Level 
to  Which 
Assigned 

Statement  of  Responsibility 

Par. 

Ho. 

AFR  80-5 

4  June  1962 

AFSC 

Establish  objectives  for  system 
analysis  and  reliability  programs 
for  use  In  the  conceptual  phase. 

6.b 

AFSCR  80-1 

14  Dec.  1962 

AFSC 

Prescribe  policies  and  procedures 
necessary  to  assure  contractual 
requirements  for  reliability  In 
each  system,  subsystem,  or  equipment 
being  developed  for  the  Air  Force 
Inventory . 

5-*.(3) 

AFSC 

divisions 

and 

oenters 

Establish  and  man  a  staff  office 
for  reliability  and  related  efforts 
(per  AFSCR  23-30)  to  Insure  estab¬ 
lishment  of  uniform  policies, 
procedures,  and  programs. 

6.d 

System 

project 

offices 

(SPO's) 

Define,  for  contractual  purposes, 
an  adequate  and  comprehensive  con¬ 
tractor  reliability  program  for 
weapon  and  support  systems,  associ¬ 
ate,  and  government-furnished 
subsystems . 

T.d 

2. 1.2. 4  Implementation  of  Qualitative 
Requirements  Selection 


The  process  of  selecting  qualitative  reliability 
requirements  for  a  particular  program  is  characterized 
by  a  series  of  background  investigations,  trade-offs, 
decisions,  and  documentary  stipulations.  This  series 
can  be  summarized  as  follows: 


(1)  Acquire  an  understanding  of  standard  Air  Force 
policy  and  requirements  for  reliability. 

(2)  Determine  the  level  of  comprehensiveness  for 
the  program. 

(3)  Estimate  the  degree  of  technical  difficulty 
in  achieving  the  quantitative  requirements  of 
the  program. 

(4)  Determine  the  availability  of  resources  (e.g., 
funds)  for  the  program. 

(5)  Define  potential  problem  areas. 

(6)  Investigate  the  applicability  of  previously 
developed  requirement  instruments. 

(7)  Select  a  method  for  outlining  the-  elements  of 
required  effort. 

(8)  Outline  the  qualitative  reliability  requirements 
to  the  extent  possible. 

(9)  Determine  major  exceptions  to  standard  Air  Force 
requirements. 

(10)  Determine  the  type  of  contractual  instruments 
to  be  used. 


These  procedural  items  are  listed  in  their  general 
order  of  accomplishment  for  a  hypothetical  situation.  Eaoh 
item  is  discussed  separately  in  following  sections. 


2. 1.2. 4.1 


Standard  Air  Force  Policy 
and  requirements 


The  initial  step  in  formulating  qualitative  require¬ 
ments  is  to  determine  the  current  Air  Force  regulations 
(at  all  levels  of  authority)  and  standard  specifications 
which  govern  reliability  programs.  The  documents  listed 
in  Table  2-4,  considered  either  individually  or.  cumulatively, 
identify  areas  of  interest  for  which  qualitative  require¬ 
ments  must  be  formulated.  In  addition,  a  more  detailed 
identification  of  areas  of  interest  is  manifested  by 
MIL-R-27542H,  Reliability  Program  for  Sy8tqnB,  Subsy stems,  and 
Equipment.  Concurrently,  it  is  compulsory  that  MIL-R-&7542A  be 
incorporated  in  all  system,  subsystem,  and  equipment  contracts. 


SPO  Reliability  Coordinators  should  attain  a  complete 
understanding  of  the  content  of  the  documents  mentioned 
above  (or  their  successors)  since  these  documents  con¬ 
stitute  directives  for  the  formulation  of  reliability 
requirements. 


2. 1.2. 4. 2  Level  of  Comprehensiveness 
for  Program 


There  is  no  precise  method  for  objectively  stipulat¬ 
ing  the  level  of  comprehensiveness  which  will  best  serve 
the  interest  of  a  new  program.  The  scope  of  the  program 
must  be  initially  estimated  on  a  relative  basis  and  sub¬ 
sequently  refined  through  a  series  of  trade-offs.  However, 
as  part  of  the  process  of  selecting  qualitative  require¬ 
ments,  the  SPO  Reliability  Coordinator  should  develop  a 
general  understanding  of  the  desired  level  of  comprehensive¬ 
ness  and  define  it,  to  the  extent  possible,  in  descriptive 
terms.  Such  action  will  undoubtedly  require  coordination 
with  higher  management. 


Some  guidance  in  determining  the  scope  of  new  programs 
is  provided  by  AFSCR  80-1,  which  states  that  comprehensive 
reliability  programs  are  not  desired  for  feasibility,  ex¬ 
ploratory  development,  and  advanced  development  studies. 
However,  this  statement  should  not  be  considered  to  imply 
that  reliability  effort  is  to  be  omitted  from  such  studies. 


Rather,  these  are  times  for  thorough  analysis  of  a  system's 
reliability  potential. 


2. 1.2.4. 3  Degree  of  Technical  Difficulty 

in  Achieving  Quantitative  Requirement 


It  is  desirable  to  relate  the  quantitative  reliability 
requirement  for  a  system  to  current  and  anticipated  capa¬ 
bilities  for  reliability  achievement,  in  order  to  indicate 
how  much  advancement  (if  any)  in  the  state-of-the-art 
must  be  obtained  through  program  efforts.  A  sincere 
attempt  should  be  made  to  objectively  assess  the  degree 
of  difficulty  of  achieving  the  requirement  —  by  com¬ 
parison  with  past  performance  of  similar  systems,  by 
consideration  of  the  known  capabilities  of  the  appropriate 
segment  of  industry,  or  as  a  last  resort,  by  "expert 
opinion  "•  Some  assessment  should  be  made  even  if  data 
of  desired  accuracy  are  lacking. 


2. 1.2. 4, 4  Availability  of  Resources 


To  the  customer,  the  program  resource  of  principal 
interest  is  the  funds  available  for  performing  the  in¬ 
tended  effort.  Knowledge  of  the  funds  allotted  for 
reliability  can  provide  considerable  guidance  in  deter¬ 
mining  the  qualitative  reliability  requirements  to  be 
applied  to  a  program.  The  SPO  Reliability  Coordinator 
should  therefore  possess  information  concerning  the  allo¬ 
cation  of  funds  to  the  various  interests  within  the  prog¬ 
ram.  (it  is  not  meant  to  imply  that  the  SPO  Reliability 
Coordinator  does  not  contribute  to  the  process  of  alloca¬ 
tion;  rather,  it  is  assumed  that  he  will  be  continuously 
active  in  making  the  needs  of  the  reliability  program 
known.)  (See  3*4.  4.1  for  further  discussion  of  program 
funding. ) 


2.1.2. 4.5  Potential  Problem  Areas 


The  identification  of  potential  problem  areas  relative 
to  reliability  achievement  may,  typically,  be  expected  to 
concern  any  one  of  the  following: 


(1)  An  identifiable  element  of  the  system  with 
known  poor  reliability. 

(2)  An  identifiable  element  of  the  system  with 
unknown  capabilities. 

(3)  A  known  environmental  condition  (e.g.,  high 
temperature)  which  is  significantly  deter- 
mental  to  reliability. 

(4)  A  quantitative  reliability  requirement  which 
is  of  such  magnitude  as  to  cause  difficulty 
in  demonstration. 

(5)  An  apparent  difficulty  in  respect  to  capability 
for  simulating  environments. 

(6)  A  complexity  factor  which  suggests  the  need  for 
extreme  parts  control. 

(7)  A  design  concept  which  suggests  the  need  for 
extensive  development. 


2. 1.2. 4. 6  Applicability  of  Previously  Developed 
Requirement  Instruments 


In  formulating  qualitative  requirements  for  a  new 
program,  the  SPO  Reliability  Coordinator  may  be  guided  to 
some  extent  by  requirement  documentation  (e.g.,  exhibits, 
requirements  statements,  program  plans,  reports,  etc.) 
previously  generated  in  support  of  other  system  programs. 
In  determining  the  applicability  of  these  documents  for 
purposes  of  guidance,  the  following  factors  should  be  oon- 


slderedt 

(1) 

Similarity  of  program  scope 

(2) 

Similarity  of  equipment. 

(3) 

Similarity  of  environment. 

<*) 

Similarity  of  mission. 

2.1-37 


(5)  Similarity  of  quantitative  requirements. 

(6)  Currency  of  the  document. 

(7)  Demonstrated  effectiveness  of  the  document 
as  a  requirement  instrument. 


2. 1.2. 4. 7  '  Methods  of  Outlining 

Elements  of  Required  Effort 


Several  basic  approaches  can  be  used  to  initially 
outline  the  elements  of  required  program  effort,  includ¬ 
ing  the  following  four  methods: 

(l)  Identification  of  equipment  items  —  The 
reliability  program  is  described  in  terms 
of  the  equipment  configuration,  as  illus¬ 
trated  in  Figure  2-12. 


FIGURE  2-12 

IDENTIFICATION  OF  EQUIPMENT  ITEMS 


■if 


This  approach  can  be  used  when  significantly 
variable  levels  of  effort  will  be  applicable 
to  the  different  items  within  a  system.  The 
major  disadvantage  is  the  necessary  repetition 
of  requirement  statements  to  assure  coverage 
of  all  items. 

(2)  Reference  to  definable  time  periods  within 

acquisition  phase  —  The  reliability  require¬ 
ments  are  delineated  on  the  basis  of  program 
milestones  such  as  those  listed  in  APR  80-1 
as  typical  points  in  the  cycle  of  a  program. 

Figure  2-13  illustrates  the  requirements 
structure  when  these  points  are  used  as  the 
basis  for  the  outline. 


FIGURE  2-13 


IDENTIFICATION  OF  TYPICAL  PROGRAM  MILESTONES 
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Use  of  this  method  will  facilitate  the  task 
of  monitoring  fulfillment  of  requirements, 
since  it  essentially  provides  a  time  base 
for  requirements.  A  disadvantage  of  the 
approach  is  the  necessary  repetitive  state¬ 
ment  of  standard  reliability  disciplines 
because  of  the  overlapping  of  program  develop 
msnt  phasee. 

(3)  Reference  to  a  prime  specification  —  Require 
ments  for  a  program  are  outlined  by  reference 
to  the  format  of  a  specification  which  is 
intended  to  have  prime  applicability  to  the 
program.  For  example,  if  MIL-R-27542* is  used 
the  requirements  outline  will,  in  general# 
have  the  form  illustrated  in  Figure  2-l4. 


FIGURE  2-14 

OUTLINE  OF  MAJOR  SECTIONS  WITHIN  MIL-R-27542* 


A  basic  advantage  of  this  method  is  that  it  is 
responsive  to  a  prime  source  of  direction.  It 
should  be  recognized  that  the  format  of  a 
standard  specification  may  require  appropriate 
tailoring  to  meet  the  needs  of  a  specific  pro¬ 
gram  application.  However,  it  is  possible  to 
be  responsive  to  the  specification  without 
paralleling  its  format. 


(4)  Reference  to  descriptive  areas  of  technical 
activity  --  This  method  is  exemplified  by 
the  approach  used  in  AFBM  Exhibit  60-11A, 
Reliability  Specification  for  WS107A-2,  dated 
10  October  i960.  Figure  2-15  was  derived  from 
an  outline  of  the  reliability  requirements  stated 
in  that  specification.  Each  progressive  step 
included  in  the  figure  is  increasingly  descrip¬ 
tive  of  technical  or  administrative  elements  of 
the  over-all  requirement.  The  particular  ad¬ 
vantage  of  this  method  is  that  it  is  not  only 
conducive  to  defining  elements  of  work  but  Is 
also  effective  in  developing  procedures  for 
implementation. 

No  attempt  is  made  herein  to  evaluate  the  applicability 
of  the  four  methods  discussed  above.  The  last  method  Is 
suggested  as  representing  the  most  practical  approach  in 
new  programs.  However,  all  four  methods  are  descriptive 
of  desired  areas  of  coverage  and  should  be  reflected  to 
various  extents  in  the  final  statement  of  requirements. 


2.1. 2.4.8  Outline  of  Reliability  Requirements 


When  a  basic  approach  to  outlining  qualitative  relia¬ 
bility  requirements  has  been  selected  (2. 1.2. 4. 7),  the 
requirements  outline  should  be  developed  and  described  to 
the  extent  possible.  The  knowledge  acquired  and  decisions 
made  in  the  steps  discussed  in  2, 1.2. 4.1  through  2.1.2.4.6 
will  provide  guidance  in  this  process. 


The  following  check  list  items  are  typical  of  those 
which  should  be  used  in  developing  the  detailed  outlines 
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(1)  Is  the  outline  sufficiently  responsive.  In 
content,  to  the  governing  Air  Porce  specifica¬ 
tion  (e.g.,  MIL-R-27542A)? 

(2)  Does  It  also  embrace  the  policy  of  other  appli¬ 
cable  Air  Force  regulations  (e.g.,  APR  8o-l)? 

(3)  Does  It  fully  exploit  the  available  areas  of 
reliability  technology  (see  2.3  and  Appendix,  A)? 

(4)  Are  primary  requirement  areas  (e.g.,  prediction, 
failure  reporting,  demonstration)  represented 
by  major  segments  of  the  outline? 

(3)  Is  the  outline  consistent  with  the  desired  level 
of  comprehensiveness  of  the  program,  available 
resources  for  the  program,  and  the  degree  of 
difficulty  in  achieving  quantitative  require¬ 
ments? 

(6)  Does  it  provide  for  coverage  of  potential 
problem  areas? 

(7)  Does  it  identify  specific  research  areas  (if 
such  have  been  determined  to  be  necessary)? 

As  the  outline  is  developed,  it  is  good  practice  to 
stipulate  requirement  statements  which  correspond  to 
the  various  segments  of  the  outline.  (The  process  of  pre¬ 
paring  a  final  statement  of  all  requirements  is  discussed 
In  detail  In  2.1.3.)» 


2. 1.2. 4. 9  Exceptions  to  Standard  Requirements 

If  it  Is  intended  that  the  prime  contractual  instrument 
for  reliability  will  be  an  existing  standard  specification 
(e.g.,  MIL-R-27542& or  its  successor),  each  requirement  statement 
within  the  standard  should  be  evaluated  for  applicability  to  the  specific 
program. 
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2.1.2.4.10  Type  of  Contractual  Instrument^  s) 


There  are  several  approaches  that  can  be  taken  to 
introduce  qualitative  reliability  requirements  into  a 
contract  for  a  particular  program,  including: 


(1)  Establishing  an  existing  standard  (e.g., 
MIL-R-27542*  or  its  successor)  as  the  sole 
statement  of  requirements  and  noting  any 
exceptions  thereto. 

(2)  Initiating  an  exhibit  (e.g.,  a  document 
similar  to  AEBM  60-11A;  which  will  serve 
as  the  standard  for  the  program. 

(3)  Preparing  a  statement  of  work  which  in¬ 
corporates  all  statements  and  references 
representing  reliability  requirements. 


©le  title  of  the  ultimate  contractual  instrument 
for  stating  program  reliability  requirements  may  vary, 
but  the  following  are  essential  factors  in  planning 
the  contract  format: 


(1)  Reliability  should  be  considered  as  a  distinct 
discipline  within  the  contract. 

(2)  If  possible,  all  reliability  requirements  should 
be  contained  by  direct  statement  or  reference 
within  one  identifiable  document. 

(3)  The  statement  of  requirements  should  include 
both  numerical  reliability  requirements  with  confi¬ 
dence  levels  (e.g.,  an  achieved  probability  of  mission  success) 
and  qualitative  requirements  which  describe 

the  desired  elements  of  work  or  level  of  effort. 


2.1.2. 5  Problems  and  Special  Considerations  Relating  to  Qualitative 
Requirements  Selection 

2. 1.2. 5.1  State  of  Development  of  System  Concept 

At  the  outset  of  any  new  program,  there  will  usually  he  some 
relative  estimate  of  the  degree  of  state-of-the-art  advancement 
required  to  achieve  the  program  objective.  Selected  qualitative 
requirements  should  reflect  the  advancement  required.  If  the 
proposed  program  represents  significantly  new  design  concepts, 
the  requirements  should  emphasize  those  techniques  which  are  most 
actively  pursued  during  development  (e.g.,  prediction,  apportionment, 
design  revie^l  If  the  program  is  primarily  a  production  contract 
for  a  large  number  of  items,  emphasis  should  be  placed  on  maintain¬ 
ing  quality.  If  the  program  includes  field  or  service  evaluation, 
emphasis  should  be  placed  upon  field  failure  reporting  and 
resulting  analysis. 

2. 1.2. 5-2  Type  of  Equipment  Involved 

Qualitative  requirements  should  be  tailored  to  the  particular 
type  of  equipment  involved  in  the  program.  For  example,  a  system 
with  an  enormous  number  of  electronic  parts  (e.g.,  a  communications 
satellite)  should  be  supported  by  a  reliability  program  which 
emphasizes  part  improvement ,  selection,  and  control.  If  the  system 
involves  propulsion,  special  emphasis  should  be  given  to  the  develop¬ 
ment  of  prediction  techniques  for  mechanical  configuration.  Or, 
if  the  program  involves  development  of  ground  equipments,  emphasis 
could  be  placed  on  the  employment  of  redundancy  (since  weight  and 
size  t'.rc  vraally  less  critical  in  such  equipments)  or  on  maintain¬ 
ability  concepts. 

2. 1.2. 5. 3  Continuity  of  Requirements  Formulation 

The  formulation  of  qualitative  reliability  requirements  is 
a  process  of  continual  development:  each  succeeding  step  in  deriving 
requirements  represents  a  refinement  or  amplification  of  preceding 
effort.  The  continuity  of  the  process  is  hindered  when  a  program 
plan  Is  prepared  without  basic  reference  to  contract  requirements 
or  contract  requirements  are  prepared  without  references  to  governing 
specifications .  In  such  instances,  it  becomes  difficult  to  relate 
and  evaluate  program  documents  in  respect  to  higher  order  of 
precedence,  and  the  terminology  in  the  documents  is  often  inconsistent . 
Pull  reference  to  controlling  documents  should  he  nmde  in  all  program 
plans  and  contracts. 
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2. 1.2. 6  Additional  References  on  Qualitative 


Requirements  Selection 

(1)  General: 

Military  Manag&aent  of  Missile  Quality  Control/ 
Programs,  R.  W.  Stalley,  Proceedings,  Ninth 
National  Symposium  on  Reliability  and  Quality 
Control,  January  1963,  PP-  66-68. 

Analysis  of  Reliability  Management  in  Defense 
Industries,  V.  J.  Bracha,  BSD-TBR-62-48, 

June  1962. 

(2)  Air  Force  Regulations,  Specifications,  Exhibits: 

MIL-R- 26484A,  Reliability  Requirements  for 
Development  of  Electronic  Subsystems  or  Equipment, 
18  April  I960.  1 

MIL-R-27542A,  Reliability  Program  for  Systems, 
Subsystems,  and  Equipment,  ^1  May  1963. 

AFBM  Exhibit  60-11A,  Reliability  Specification 
for  WS107A-2,  10  October  I960  (see  Appendix  C). 


2.1.3  Preparation  of  Statement 

of  Reliability  Requirements 


2. 1.3.1  Definition 


Onoe  the  quantitative  and  qualitative  reliability 
requirements  for  a  program  have  been  determined  (see 
2.1.1  and  2.1.2),  a  formal  statement  of  the  requirements 
must  be  prepared  for  ultimate  Incorporation  Into  a  con¬ 
tract.  The  Importance  of  clarity  and  completeness  of 
this  requirements  statement  Is  emphasized  by  several 
considerations:  It  represents  the  Initial  effort  for 
describing  the  manner  In  which  reliability  tasks  are  to 
be  accomplished;  It  represents  the  dooument  whloh  must 
be  used  by  bidders  in  describing  their  plan  and  estimat¬ 
ing  the  00s t  of  the  work  to  be  performed;  and  It  also 
represents  the  standard  upon  which  the  bidders  will  be 
evaluated . 

The  activity  discussed  In  2.1.3  Is  defined  as  that 
effort  necessary  to  amplify,  refine,  or  otherwise  adapt 
the  seleoted  reliability  requirements  so  as  to  express 
them  In  a  form  suitable  for  Inclusion  In  a  procurement 
contract.  This  effort  commences  when  an  outline  of  re¬ 
quirements  has  been  formulated  and  continues  until  a 
formal  statement  which  oan  serve  as  a  oontraotual  instru¬ 
ment  has  been  prepared. 


2. 1.3. 2  Mr  Porce  Policy 


Mr  Foroe  policy  whloh  relates  to  preparation  of 
oontraotual  Instruments  for  reliability  requirements  Is 
summarised  In  Table  2-6. 


2. 1.3. 3  Responsibilities  for  Preparation 
of  Requirements  Statement 


The  responsibilities  of  SPO  Reliability  Coordinators 
and  other  assoolated  levels  of  authority  relating  to  pre¬ 
paration  of  oontraotual  Instruments  for  reliability  re¬ 
quirements  are  summarised  In  Table  2-7. 


TABLE  2-6 


SUMMARY  OP  AIR  FORCE  POLICY 
RELATING  TO  PREPARATION  OP  CONTRACTUAL  INSTRUMENTS 
FOR  RELIABILITY  REQUIREMENTS 


Air  Force 
Document 

Statement  of  Policy 

Par. 

No. 

APR  80-5 

4  June  1962 

(l)  Reliability  will  be  stressed  during  early 
system  studies ,  source  selection,  design, 
development,  and  production. 

4.a.(l) 

(2)  During  the  conceptual  and  acquisition 
phases,  systems  will  be  analyzed  and  a  relia¬ 
bility  program  established  for  each.  Program 
will  include  a  minimum  acceptable  reliability 
level  as  well  as  a  reliability  goal,  with 
intermediate  quantitative  values  required  to 
measure  progression.  Where  sufficient  sampl¬ 
ing  permits,  a  stated  minimum  acceptable  con¬ 
fidence  level  should  be  included  for  each 
probability  value. 

4.a.(3) 

(3)  Re procured  spares  and  parts  will  be  pro¬ 
cured  to  definitive  reliability  requirements. 

4. a. (4) 

(4)  Specifications,  exhibits,  work  statements, 
product  descriptions,  and  contracts  for  sys¬ 
tems  and  associated  materiel,  including  GFE 
for  inventory,  will  include  specific  minimum 
acceptable  reliability  requirements  as  one  of 
the  major  engineering  factors. 

4.b 

(5)  Systems  contracts  will  include  a  require¬ 
ment  for  a  comprehensive  contractor  relia¬ 
bility  program,  including  quantitative  require¬ 
ments  and  tests  for  reliability  demonstration. 

4.  c 

APR  80-5 

4  June  1962 
AFSCR  80-1 

14  Dec.  1962 

(l)  Individual  parts  specifications  will  in¬ 
clude  current  failure  rate  level  and  up  to  four 
discrete  graduated  levels  representing  state- 
of-the-art  advancements.  The  total  number  of 
levels  will  be  governed  by  system  requirements. 

4.b/ 

4.d 
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TABLE  2-6  (Continued) 

SUMMARY  OP  AIR  FORCE  POLICY 
RELATING  TO  PREPARATION  OP  CONTRACTUAL  INSTRUMENTS 
FOR  RELIABILITY  REQUIREMENTS 


Air  Force 
Document 

Statement  of  Policy 

Par. 

No. 

AFSCR  80-1 

14  Dec.  1962 

(l)  Quantitative  reliability  requirements 
will  be  developed  from  stated  objectives 
and  stated  in  specific  numerical . terms  in 
the  appropriate  contractual  documents  and 
systems  specifications  will  inolude  the 
definition  of  satisfactory  operation,  in¬ 
cluding  operating  time  or  cycles,  environ¬ 
mental  conditions, and  where  practicable, 
the  confidence  levels  to  which  specified 
probabilities  will  be  demonstrated. 

4.c 

(2)  Quantitative  reliability  figures  will 
be  stipulated  for  the  reliability  goal, 
the  minimum  aooeptable  reliability  require¬ 
ment,  and  the  achieved  reliability  at  such 
specified  intermediate  points  as  necessary 
to  measure  the  progress  of  the  reliability 
effort . 

4.d 

(3)  Contracts  for  aerospace  systems,  major 
subsystems,  and  equipment  will  Include  a 
requirement  for  a  comprehensive  and  organ¬ 
ized  contractor  reliability  program  that 
extends  through  subcontractor  and  vendor 
levels . 

U.e 

(4)  Integrated  test  plans  to  investigate 
causes,  effects,  modes  of  failure,  and  to 
demonstrate  achieved  reliability  will  be 
developed  and  incorporated  in  contractual 
documents.  Plans  will  be  designed  to  pro¬ 
vide  the  maximum  information  and  assuranoe 
consistent  with  the  state-of-the-art. 

4.f 

(5)  Specific  activities  may  be  identified 
as  reliability  effort  in  order  to  provide 
a  management  tool  for  guiding  and  assessing 
adequaoy  of  effort. 

U.g 

AFBSUR  80-5 

28  Dec.  1962 

MIL-R-27542  (or  superseding  publications) 
quantitative  requirements,  and  provisions 
for  demonstration  will  be  incorporated  into 
future  Air  Force  balllstlo  missile  oontraete 
and  into  existing  oontraots  as  appropriate. 

5-c 

2.1-49 


TABLE  2-7 

SUMMARY  OP  RESPONSIBILITIES 
RELATING  TO  PREPARATION  OP  CONTRACTUAL  INSTRUMENTS 
FOR  RELIABILITY  REQUIREMENTS 


Air  Poree 
Document 

Level 
to  Which 

Assigned 

Statement  of  Responsibility 

APR  80-5 

4  June 
1962 

APSC 

(1)  Incorporate  quantitative  reliability 
requirements  and  provisions  for  demon¬ 
strating  reliability  in  all  specifications, 
exhibits,  product  description,  work  state¬ 
ments  and  contractual  clauses  to  be  ref¬ 
erenced  to  or  Included  in  contracts  for 
systems  and  associated  material,  Hiese 
documents  will  include  specific  provisions 
related  to  procurement,  production,  and 
quality  control  processes  necessary  to  in¬ 
sure  achievement  of  the  required  relia¬ 
bility. 

6.  c 

(2)  Assume  responsibility  for  system  pro¬ 
curement  reliability  activities,  including 
quality  control,  production,  packaging, 
transportation,  and  storage. 

6-g 

APLC 

(1)  Assume  responsibility  for  logistic 
support  reliability  activities  as  related 
to  procurement,  production,  packaging, 
transportation,  supply,  maintenance, 
quality  control,  and  materiel  management. 

T-b 

APSCR  80-1 

14  Dec. 
1962 

APSC 

(DCS/Sya terns) 

(l)  Prescribe  policies  and  procedures 
necessary  to  assure  contractual  require¬ 
ments  for  reliability  in  each  system, 
subsystem,  or  equipment  being  developed 
for  the  Air  Poree  inventory. 

5-a.(3) 

APSC 

(DCS/ 

Procurement 

and 

Materiel) 

(1)  Prescribe  procurement  policies  for 
Inclusion  of  reliability  requirements  in 
oontraots  and  invoke  monetary  penalties, 
unit  prloe  decreases,  or  other  considera¬ 
tions  deemed  equitable  if  requirements 
are  not  met. 

5-b.(l) 

System 

Program 

Offices 

(SPO's) 

(1)  Incorporate  reliability  requirements, 
including  provisions  for  demonstrations  of 
reliability  in  all  specifications,  exhibits, 
produot  descriptions,  or  other  contractual 
documents  to  be  referenced  in  contracts  for 
weapon  and  support  systems  and  associated 
material  (including  OPE).  Prime  contrac¬ 
tors  will  be  required  to  Impose  adequate 
requirements  on  subcontractors  and  vendors. 
Contractual  documents  will  lnolude  any 
speolflo  requirements  related  to  the  con¬ 
trol  of  manufacturing  processes  necessary 
to  Insure  the  desired  reliability. 

7-c 

(2)  Define  for  contractual  purposes  an 
adequats  and  comprehensive  contractor 
reliability  program  for  weapon  and  support 
systems ,  associate  and  government- furnished 
subsystems. 

7-d 

(3)  Incorporate  requirements  for  quanti¬ 
tative  data  in  contractual  decuman ts , 

7-e 

APBSDR  60-5 
26  Dec. 
1962 

3P0 

Reliability 

Coordinators 

(1)  Assist  the  OPR  for  reliability  in 
developing  reliability  specifications, 
policies,  and  procedures  in  accessing  the 
over-all  BSD  reliability  program  effec¬ 
tiveness. 

6.b.(5) 

2.1.3.**  Implementation  of  Preparation 
of  Requirements  Statement 


Two  primary  considerations  govern  the  preparation  of 
a  statement  of  reliability  requirements  for  a  specific 
program: 

(1)  The  statement  should  be  directly  tailored  to 
the  needs  of  the  program.  --  The  most  Instru¬ 
mental  factor  In  determining  the  elements  of 
work  to  be  performed  and  the  necessary  level 

of  effort  Is  the  numerical  reliability  require¬ 
ment.  Hence,  If  the  numerical  requirement  for 
the  system  Is  conceived  on  the  basis  of  the 
needs  of  the  program  (as  should  be  the  case). 

It  should  follow  that  the  details  of  the  work 
required  will  also  be  based  on  those  needs. 

(2)  The  requirements  should  be  stated  in  as  much 
detail  as  possible.  —  In  preparing  standards 
which  are  Intended  for  use  in  a  large  number 
of  programs,  considerable  flexibility  must  be 
maintained  In  setting  forth  requirements. 

Hence,  when  such  general  standards  are  applied 
to  particular  programs,  they  must  be  augmented 
by  requirements  statements  specifically  directed 
to  the  needs  of  the  given  systems.  High  Air 
Force  authority  has  stated:  "In  the  specifica¬ 
tion  of  reliability  requirements.  It  Is  to  the 
advantage  of  Industry  for  government  agencies 

to  be  as  definitive  as  possible.  It  Is  our 
objective  to  continue  to  eliminate  vagueness 
and  generality  from  reliability  requirements."* 

Reliability  requirements  stated  in  a  contract  may  be 
subject  to  later  amplification,  refinement,  or  even  change. 
However,  anticipation  of  any  such  actions  subsequent  to 
oontrapt  award  should  not  lessen  the  objective  of  attaining 
olarlty  and  completeness  In  the  Initial  contractual  state¬ 
ment  of  requirements. 


•major  General  0.  J.  Rltland,  Commander  Space  Systems 
Division,  U.  S.  Air  Foroe,  In  paper  entitled,  "Specify¬ 
ing  Reliability  in  Military  Contracts,"  presented  at 
Seventh  Military-Industry  Missile  and  Space  Reliability 
Sympeelum,  June  1962. 
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2.1. 3.4.1  Format  of  Requirements  Statement 


In  selecting  a  documentary  format  for  inclusion  of 
reliability  requirements  within  a  contract,  the  primary 
considerations  are  that  the  requirements  be  separately 
identified  and  be  contained  within  one  continuous  section 
of  the  over-all  contract.  The  descriptive  title  given  to 
the  statement  of  requirements  (e.g.,  "Work  Statement," 
"Exhibit,"  etc.)  is  of  less  importance  and  will  vary  from 
contract  to  contract. 


2. 1.3. 4.2  Content  of  Requirements  Statement 


The  content  of  the  statement  of  requirements  will, 
of  course,  vary  with  each  individual  system.  However, 
the  following  outline  is  a  general  guide  to  statement 
content : 

(1)  Scope  —  A  brief  statement  of  the  system  to 
which  the  requirements  are  applicable,  the 
prime  specification  (e.g.,  MIL-R-27542A)  to 
which  the  requirements  are  responsive,  and 
any  other  information  pertinent  to  the 
coverage  intended . 

(2)  Applicable  Documents  --  A  complete  listing 
of  the  government  (or  other  documents)  which 
govern  the  requirements,  including  designation 
of  the  appropriate  Issues  and  statement  of 
the  order  of  precedence  of  the  documents. 

(3)  Requirements  —  A  complete  listing  and  descrip¬ 
tion  of  the  qualitative  and  quantitative  re¬ 
quirements  which  have  been  established  for  the 
reliability  program. 

(4)  Definitions  —  Brief  explanations  of  the  intended 
meanings  of  terms  unique  to  the  statement  of 
requirements . 


2. 1.3. 4. 3  Relation  of  Requirements  Statement 
to  Other  Contractual  Documents 


Most  system  procurements  will  require  conformance  to 
both  a  statement  of  reliability  requirements  specifically 
designed  for  the  program  and  at  least  one  standard 
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specification  (e ,g.#  MIL-R-27542A)  of  more  general  applica¬ 
tion.  The  relationship  between  these  various  contractual 
documents  must  be  clearly  established.  The  specific  state¬ 
ment  of  requirements  will  most  likely  stipulate  additional 
tanks  as  well  as  interpret  certain  of  the  tasks  directed  by 
the  general  speoifloation.  Care  must  be  taken  that  the 
"additional"  tasks  and  "interpretive"  tasks  are  distinctly 
reoognized,  so  that  the  specific  program  document  will  not 
contribute  to  duplication  of  effort.  If  the  "additional" 
tasks  are  Intended  to  replace  similar  efforts  described  In 
the  general  specifications,  proper  exceptions  should  be 
noted  In  the  program  document. 


2.1.3.^.^  Definitiveness  of  Requirements  Statement 


In  preparing  the  statement  of  reliability  requirements 
for  a  specific  program,  maximum  use  should  be  made  of  defin¬ 
itive  words  and  phrases.  For  example,  words  suoh  as  "period¬ 
ically"  or  continuously"  should  be  replaced  by  "monthly," 
"quarterly,"  "30  days  after  contract  award,"  or  other  time- 
definitive  phrases.  When  reliability  tasks  (e.g.,  prediction, 
apportionment)  are  described,  the  number  of  times  or  frequency 
of  performance  should  be  stipulated  where  possible.  General 
statements  must  be  used  In  some  cases  because  the  precise 
level  of  effort  or  scope  of  work  will  not  be  known  at  the 
time  contract  requirements  are  prepared.  However,  their  use 
should  be  minimized  to  the  extent  possible. 

The  SPO  Reliability  Coordinator  can  find  some  guidance 
In  respect  to  format,  content,  deflnltlveness,  and  other 
considerations  In  preparing  a  requirements  statement  for  a 
particular  program  by  comparing  the  statements  provided  for 
other  similar  programs.  Sample  reliability  work  statements 
are  presented  In  Appendix  C  to  Illustrate  ways  in  which 
reliability  requirements  can  be  specified  for  system  pro¬ 
grams. 


2. 1.3. 4. 5  Further  Definition  of  Requirements 
Statement  by  Contractor 


It  is  usually  desirable  that  requirements  statement 
preparation  be  as  complete  as  possible  prior  to  the  aotual 
initiation  of  oontraot  documentation.  In  some  instances, 
however,  the  procuring  aotivity  may  allow  the  contractor,  as 
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part  of  his  effort,  to  describe  requirements  more  fully  and 
accurately;  however,  the  tentative  plans  of  the  contractor  must 
be  stated  so  that  the  customer  can  approve  them  prior  to 
issuance  of  the  contract.  Seme  negotiation  details  may  be 
allowed  after  contract  award,  but  these  details  should  be  defined 
explicitly  and  documented  prior  to  contract  award. 

2.1.3«?  Problems  and  Special  Considerations 
Relating  to  Preparation  of 
Requirements  fltatSBant 


2.1. 3-5.1  Statement  of  Demonstration 
Requirements 


Difficulty  is  frequently  experienced  in  stipulating  the 
requirements  for  demonstrating  achieved  reliability,  particularly 
high  quantitative  reliability  requirements  such  as  those  normally 
associated  with  missile  and  space  programs .  For  example,  a 
communications  satellite  with  a  required  mean  time  to  failure  of 
one  year  would  require  over  20,000  hours  of  failure-free  operation 
in  order  to  demonstrate  attainment  with  90 %  statistical  confidence. 
Or,  a  one-shot  device  with  a  required  probability  of  survival 
of  0.90  would  require  14  successes  out  of  15  trials  to  demonstrate 
attainment  at  90?  confidence.  Whan  the  typical  costs  associated 
with  test  items  are  considered,  it  is  apparent  that  the  expense 
of  tests  to  demonstrate  achievement  of  high  reliability  require¬ 
ments  could  assume  great  proportions. 

There  are  no  routine  solutions  to  the  problem  of  lowering 
the  costs  of  demonstration  teats,  but  there  are  several  consid¬ 
erations  which  may  facilitate  accomplishing  the  purposes  of 
demonstration : 

(l)  Extend  the  period  of  demonstration  beyond  early  hardware 
delivery  dates  and  stipulate  that  demonstration  be 
aade  at  the  latest  possible  time  when  it  can  be  tolerated. 


(2)  Permit  contractors  to  satisfy  reliability 
requirements  by  reference  to  previous  tests 
conducted  on  their  products. 

(3)  Permit  demonstration  tests  which  acoept  a 
lower  order  of  confidence. 

(4)  Permit  use  of  accelerated  test  methods  where 
practicable . 

The  preceding  discussion  Is  not  Intended  to  Imply 
that  the  objectives  of  demonstration  should  be  compromised 
or  that  Insurmountable  problems  will  always  be  encountered. 
Rather,  It  Is  Intended  to  convey  that  the  subject  of 
demonstration  should  be  aotlvely  considered  during  the 
requlrementa  formulation  phase.  In  this  way,  any  potential 
problems  whloh  may  characterize  the  program  can  receive  the 
full  benefit  of  management  consideration  at  the  earliest 
possible  time  In  the  program,  and  maxamua.  time  for  solution 
of  these  problems  can  be  provided. 


2. 1.3.6  Additional  References  on  Preparation 
of  Requirements  Statement 


Reliability:  Manager 
D.  K.  Lloya  ana  n.  Lii 

pp.  20-26. 


Reliability  Principles  imd  Practices, 

S.  R.  Calaoro,  Mcoraw-Hill,  190^*  pp.  239-255. 

Reliability  Procurement.  M.  R.  Seldon, 
Industrial  quality  control ,  vol .  17 , 

April  1961#  pp.  8-9. 

Rsssaroh  and  Development  Reliability,  R.  H. 
if  41)7  Technical  Publications  Committee , 
Sleotronlos  Division  of  Amerloan  Soolety  for 
Quality  Control,  1961. 


Meyers 


wm  wMraiumit, 

and  Quality  Control,  January  1959*  PP*  246-250. 


O 
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2.2  Selection  of  Contractor 


Air  Force  directives  unequivocally  require  that 
quantitative  and  qualitative  reliability  commitments 
be  featured  in  every  procurement  contract  for  the 
development  of  aerospace  items  (see  Tables  2-1  and 
2-4). 


Implementation  of  this  basic  requirement  necessi¬ 
tates  important  preparatory  activities  on  the  part  of 
the  System  Program  Office:  First,  system  reliability 
requirements  must  be  determined  (2.1.1  and  2.1.2)  and 
formally  documented  (i.e.,  preparation  of  a  specific 
contractual  work  statement  as  discussed  in  2.1.3  and 
compilation  of  other  documents,  such  as  military  speci¬ 
fications,  which  set  forth  general  requirements).  Then, 
a  contractor  must  be  selected  who  has  the  necessary 
capability  to  achieve  the  system  requirements .  System 
Program  Office  activities  which  pertain  directly  to 
selection  of  a  contractor  are  discussed  in  this  section. 

In  order  for  adequate  contractor  evaluation  to  be 
performed,  the  selection  activities  should  begin  as  early 
in  the  program  as  possible.  The  major  areas  of  activity 
(and  the  sections  in  which  they  are  discussed)  are: 

(1)  Selection  of  qualified  bidders  (2.2.1). 

(2)  Preparation  of  the  request  for  proposal  (rfp)  (2.2.2). 

(3)  Evaluation  of  proposals  (2.2.3).  Not  all 
bidders  will  respond  to  the  program  reliability 
requirements  with  equally  adequate  proposals. 

Some  standard  means  must  therefore  be  employed 
to  evaluate  each  proposal  for  adequacy. 

(4)  Negotiation  of  the  contract  (2.2.4).  This 
seemingly  routine  activity  provides  an  oppor¬ 
tunity  for  the  customer  to  assure  that  all 

of  his  requirements  are  covered  by  contractual 
obligations  upon  the  contractor. 
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2.2.1  Selection  of  Qualified  Bidders 


2. 2. 1.1  Definition 


The  activity  of  selecting  qualified  bidders  involves 
an  analysis  and  evaluation  process  to  determine  which  of 
the  many  potential  contractors  are  more  likely  to  fulfill 
efficiently  the  reliability  and  other  requirements  of  a 
system  program. 

The  potential  bidders  are  evaluated  on  the  basis  of 
past  performance  and  manifest  capabilities.  Capability 
to  meet  reliability  requirements  is  one  of  several  con¬ 
siderations  in  evaluating  bidders ,  but  it  is  a  vital 
criterion  in  respect  to  ultimate  program  success  and 
should  receive  equal  emphasis  with  other  important  con¬ 
tractor  qualifications . 


2. 2.1. c  Air  Force  Folicy 


Air  Force  policy  pertaining  to  reliability  consldera-  3 

tlons  in  selection  of  qualified  bidders'  is  summarized  in 
Table  2-8. 


2. 2. 1.3  Responsibilities  for  Selection 
of  Qualified  Bidders 


The  assigned  responsibilities  of  SPO  Reliability 
Coordinators  and  other  associated  levels  of  authority 
relating  to  selection  of  qualified  bidders  are  summarized 
In  Table  2-9.  These  general  SPO  responsibilities  can, 
in  turn,  be  interpreted  to  imply  more  explicit  responsi¬ 
bilities,  as  follows: 

(1)  To  carry  out  all  Air  Force  regulations  pertain¬ 
ing  to  reliability  assurance. 

(2)  To  stress  reliability  in  the  selection  of  a 
source  for  items  or  services. 

(3)  To  consider  a  contractor’s  reliability  capa¬ 
bility  (as  manifest  In  past  performance)  as 

a  major  factor  In  all  source- selection  action. 
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TABLE  2-8 

SUMMARY  OP  AIR  FORCE  POLICY 
RELATING  TO  RELIABILITY  CONSIDERATIONS 
IN  SELECTION  OP  BIDDERS 


Air  Force 
Document 

Statement  of  Policy 

Par. 

No. 

AFR  80-5 

b  June  1962 

(1)  Reliability  will  be  stressed  during 
early  system  studies,  source  selection, 
design,  development,  and  production.  Its 
adequacy  depends  upon  the  emphasis  received 
during  the  conceptual  and  acquisition 
phases. 

(2)  Contractor's  reliability  capability, 
considering  both  past  performance  and 
proposed  programs,  will  be  a  major  factor 
in  all  source  selection  action. 

U.a.(l) 

4.e 
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TABLE  2-9 

SUMMARY  OP  RESPONSIBILITIES 
RELATING  TO  RELIABILITY  CONSIDERATIONS 
IN  SELECTION  OF  BIDDERS 


Air  Force 
Document 

Level 
to  Which 
Assigned 

Statement  of  Responsibility 

Par. 

No. 

APR  80-5 

4  June  1962 

APSC 

(1)  Include  specific  provisions 
related  to  procurement*  produc¬ 
tion*  and  quality  control  pro¬ 
cesses  neoessary  to  insure 
achievement  of  the  required 
reliability. 

6.c 

(2)  Determine  the  adequaoy  of 
eaoh  Air  Poroe  contractor's 
reliability  program  for  achieve 
lng  and  demonstrating  Air  Poroe 
produot  reliability  goals. 

6.  d 

(3)  Assume  responsibility  for 
system  procurement  reliability 
activities . 

6.g 

APS OR  80-1 

lh  Dec.  1962 

APSC  DCS/ 
Procurement 
and.  Materiel 

Prescribe  procurement  policies 
for  inclusion  of  reliability 
requirements . 

5.b.(l) 

AFBSDR  80-5 

26  Dec.  1962 

SPO 

Reliability 

Coordinators 

Assume  responsibility  for  all 
aspeots  of  the  System  relia¬ 
bility  program  including  all 
aspeots  of  the  contractor 's 
reliability  program. 

6.  b.(3) 

(4)  To  determine  the  adequacy  of  each  potential 
contractor’s  reliability  program  for  achieving 
Air  Force  product  reliability  goals. 

(5)  To  assume  responsibility  for  all  reliability 
procurement  activities. 

In  summary,  a  SPO  Reliability  Coordinator  is  responsible 
for  taking  all  possible  steps  to  assure  that  bidders  are 
selected  on  the  basis  of  their  inherent  capability  to 
produce  a  product  having  the  required  level  of  reliability. 


2. 2. 1.4  Implementation  of  Selection 
of  Qualified  Bidders 


2. 2. 1.4.1  Knowledge  of  Reliability  Requirements 


By  the  time  selection  of  bidders  becomes  a  factor 
of  immediate  interest  in  a  program,  all  Bystem  reliability 
requirements  should  have  been  established  (see  2.1.1, 
2.1.2,  and  2.1.3).  The  SPO  Reliability  Coordinator,  be¬ 
cause  of  his  contribution  to  the  formulation  of  these 
requirements,  should  be  thoroughly  familiar  with  their 
content  and  thus  understand  the  Bcope  and  level  of  effort 
which  a  contractor  must  implement  to  fulfill  the  require¬ 
ments  . 


2. 2. 1.4. 2  Sources  of  Bidder  Information 


Potential  bidders  can  be  found  in  Approved  Vendor 
lists  and  other  lists  having  a  similar  purpose.  In 
general,  such  lists  are  not  exhaustive  of  all  contractors 
capable  of  performing  the  desired  work.  Efforts  should 
therefore  be  made  to  determine  as  many  additional  potential 
suppliers  as  possible  for  preliminary  evaluation.  The 
potential  contractor  nonchalantly  overlooked  may  be  the 
very  one  most  oapable  of  providing  reliable  equipment. 


2. 2. 1.4. 3  Preliminary  Evaluation  of  Bidders 


The  gross  list  of  potential  bidders  should  be  analyzed 
for  correlation  between  obvious  capabilities  and  the  pro¬ 
gram  needs.  Entries  having  unacceptable  correlations  should 
be  eliminated. 


All  potential  bidders  should  be  compared  on  an  equal 
basis  in  this  preliminary  analysis.  A  check  list  such  as 
that  shown  in  Figure  2-io  or  one  tailored  for  the  particular 
program  should  be  used.  If  a  standard  bidder  evaluation 
check  list  is  developed  and  used,  it  may  be  adapted  to  dif¬ 
ferent  program  requirements  by  varying  the  emphasis  placed 
on  the  items  in  the  list  (e.g.,  "weighting  factors"  can  be 
applied  to  guide  the  evaluator  as  to  the  relative  importance 
of  specific  items). 

The  check  list  of  Pigure  2-l6  is  designed  so  that  the 
information  necessary  to  its  completion  can  be  obtained  by 
either  direct  contact  or  telephone  response  to  the  questions 
listed.  It  is  entirely  appropriate  to  visit  the  facilities 
of  a  potential  bidder  to  survey  his  facilities  and  to  talk 
to  his  personnel .  If  essential  information  regarding  a 
potential  bidder  is  lacking,  a  reasonable  effort  should  be 
made  to  obtain  it  before  the  entry  is  eliminated.  Every 
qualified  source  of  materials,  equipments,  or  services  is 
a  valuable  asset  to  the  Air  Force  in  meeting  its  rapidly 
and  widely  expanding  technological  needs. 

It  is  not  necessary  to  perform  a  oomplete  evaluation 
of  the  capabilities  of  potential  bidders  in  order  to  deter¬ 
mine  a  list  of  qualified  bidders,  The  effort  in  this  pre¬ 
liminary  evaluation  is  to  determine  bidders  who  can  perform 
adequately  and  who  should  therefore  be  given  an  opportunity 
to  submit  proposals  for  the  planned  program.  Bidders  whose 
technical  proposals  and  reliability  program  plans  are 
aooeptable  (see  2.2.3)  will  be  subjected  to  thorough  ex¬ 
amination  before  a  oontraot  is  let  (see  2.2,4). 


2. 2. 1.5  Problems  and  Special  Considerations 

Relating  to  Selection  of  Qualified  Bidders 


2. 2. 1.5, 2  Inadequate  Information 

on  System  Reliability  Requirements 


Laok  of  knowledge  regarding  the  aotual  requirements 
of  the  aystem  or  produot  may  lead  to  subjectiveness  in 
■eleotlon  of  bidders.  The  evaluator  may  place  emphasis 
on  oertaln  requirements  and  arrive  at  one  list  of  potential 
bidders,  while  emphasis  on  other  requirements  would  result 
in  an  entirely  different  list  of  perhaps  better  qualified 
bodders. 
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FIGURE  2-16 

SAMPLE  CHECK  LIST  FOR  DETERMINING  QUALIFIED  BIDDERS 


BIDDER  RELIABILITY  EVALUATION  QUESTIONNAIRE 

Company 

Program 

Address 

Considered  for 

Date 

To  what  corporation  officer 
does  the  reliability 
organization  report? 

President  _ 

Corporate  Vice  President 
Quality  Assurance 

Chief  Engineer 

Engineering  Subgroup 

Quality  Control  Dept.  _ 

Data  Collection 

What  are  the  distinguishable 

Data  Analysis 

areas  of  activity  shown  in 

Physics  of  FaTfure 

the  corporation  reliability 

Failure  Reporting 

organization  chart  and  how 

Parts  Control 

many  people  staff  each  group? 

Statistical  Methods 

Reliability  Test  Lab. 

Environmental  Test  LaFI 

Design  Review  group 

What  is  considered  to  be  the 
most  sophisticated  project 
undertaken  by  the  corporation 
to  date? 

Less  Advanced  _ 

Equally  Advanced  _ 

More  Advanced  _ 

In  what  terms  were  relia¬ 
bility  requirements  imposed 
in  that  project? 

Values  required  and  attained? 

Past  Reliability  Required _ 

Past  Reliability  Attained _ 

Have  improved  methods  or 
techniques  since  been  learned? 

Reliability  Growth 

Yes  W 

What  per  cent  of  the  previous 
project  money  was  devoted  to 

Per  oent 

reliability  effort? 

What  is  the  total  plant 
employment? 

Plant  Employment 

» 

Successful  selection  of  bidders  and  successful  oonduot 
of  the  entire  program  depend  to  a  large  degree  upon  the 
aoouraoy  with  which  the  program  reliability  requirements 
have  been  defined.  Bidder  capabilities  should  correlate 
with  these  requirements.  For  example,  if  a  system  mission 
is  to  be  five  hours  in  a  rather  moderate  environment,  a 
list  of  bidders  having  the  capability  to  develop  moderately 
sophisticated  equipment  should  be  compiled.  Contractors 
having  far  superior  capabilities  might  not  be  sought  in 
order  to  save  money  and/or  reserve  their  capabilities  for 
more  demanding  work.  If  the  mission  duration  is  to  be 
very  long,  say  seven  years,  a  group  of  bidders  having 
experience  In  the  development  and  manufacture  of  equipment 
regularly  meeting  long-life  requirements  would  be  selected. 

Or,  if  the  use  environment  is  to  Involve  severe  conditions 
such  as  high  temperature,  a  list  of  bidders  having  experi¬ 
ence  In  design  and  production  of  equipment  for  operation 
In  such  environments  would  be  selected. 


Air  Foroe  technical  requirements  necessarily  stress 
or  even  surpass  the  «tate-of-the-art  to  suoh  a  degree  that 
bidders  cannot  always  be  found  who  have  sufficient  experi¬ 
ence  in  the  technology  required  for  a  particular  program. 
When  experlenoe/requlrements  correlation  Is  low  In  all 
entries  for  the  bidders'  list,  the  procuring  activity  must 
determine  which  bidders  have  adequate  potential  for  new 
development  along  the  required  lines.  In  such  Instances, 
faotors  other  than  speolrlo  experience  (e.g.,  available 
engineering  and  management  talents,  speolal  facilities, 
experience  In  related  fields)  should  be  evaluated. 
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2.2.2  Preparation  of  Requests  for  Proposal  (rft) 


2. 2. 2.1  Definition 


The  Request  for  Proposal  is  the  means  by  which  the 
Air  Force  communicates  its  need  for  a  product  or  service 
to  potential  contractors .  It  must  describe  the  desired 
product  or  service  in  sufficient  detail  so  that  bidders 
can  reasonably  determine  whether  they  have  the  interest 
and  capability  for  meeting  the  Air  Force  need.  The 
Request  for  Proposal  is  sent  to  qualified  bidders  suf¬ 
ficiently  in  advance  of  the  formal  start  of  a  program  for 
the  bidders  to  develop  proposals  in  response  and  transmit 
them  to  the  procuring  agency's  designated  office. 

The  discussion  in  2.2.2  is  primarily  concerned  with 
the  responsibilities  of  the  SPO  Reliability  Coordinator 
in  assuring  that  reliability  requirements  are  considered 
in  RFP  preparation.. 


2. 2. 2. 2  Air  Force  Policy 


Explicit  policy  statements  governing  the  preparation 
of  Requests  for  Proposal  (RFP'b)  are  provided  by  Air 
Force  procurement  directives  rather  than  by  the  directives 
relating  to  reliability.  The  reliability  directives  do 
stipulate,  however,  that  all  documentation  used  in  pro¬ 
curement  for  a  specific  program  must  reflect  the  relia¬ 
bility  requirements  of  the  program. 


2. 2. 2. 3  Responsibilities  for  Preparation 
of  Request  for  Proposal 


The  responsibilities  assigned  by  Air  Force  directives 
to  SPO  Reliability  Coordinators  and  other  associated  levels 
of  authority  regarding  the  incorporation  of  reliability 
requirements  in  RFP's  are  given  in  Table  2-10. 
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TABLE  2-10 


SUMMARY  OP  RESPONSIBILITIES 
RELATING  TO  RELIABILITY  CONSIDERATIONS 
IN  PREPARATION  OP  REQUESTS  POR  PROPOSAL 


Air  Force 
Document 

APR  80-5, 

4  June  1962 


Level 
to  Which 
Assigned 


APSC 


AFSCR  80-1 
14  Dec.  1962 


System 

Project 

Offices 

(SPO’s) 


Statement  of  Responsibility 

(1)  To  incorporate  quantitative 
reliability  requirements  and  pro¬ 
visions  for  demonstrating  relia¬ 
bility  In  all  specifications, 
exhibits,  product  descriptions, 
work  statements,  and  contractual 
clauses  to  be  referred  to  or  in¬ 
cluded  in  contracts  for  systems 
and  associated  materiel.  These 
documents  will  include  specific 
provisions  related  to  procurement, 
production,  and  quality  control 
processes  necessary  to  insure 
achievement  of  the  required  relia¬ 
bility. 

(2)  To  assume  responsibility  for 
system  procurement  reliability 
activities.  Including  quality  con¬ 
trol,  production,  packaging,  trans¬ 
portation,  and  storage. 

(l)  Incorporate  reliability  re¬ 
quirements,  including  provisions 
for  demonstration  of  reliability 
in  all  specifications,  exhibits, 
product  descriptions,  or  other 
contractual  documents  to  be  refer¬ 
enced  in  contracts  for  weapon  and 
support  systems  and  associated 
materiel . 
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2. 2. 2. 4  Implementation  of  Preparation 
of  Request  for  Proposal 


In  Air  Force  procurement  of  aerospace  systems  and 
services.  Requests  for  Proposal  are  usually  sent  to  a 
limited  list  of  potential  bidders  known  to  be  particularly 
auallfled  to  provide  the  desired  systems  or  services 
(see  2.2.1).  Specific  Instructions  for  the  Issuance  of 
RFP’s  are  given  in  ASPR  3-500  and  3-802.  Further  informa¬ 
tion  is  provided  by  AFBSD  Interim  Regulation  80-7,  Source 
Selection  Board  Procedures. 


2. 2. 2. 4.1  Content  of  Request  for  Proposal 


The  RFP  basic  document  should  contain  a  complete 
and  specific  description  of  the  items  or  services  to  be 
procured,  together  with  applicable  requirements  for 
quantities  to  be  delivered,  time  and  place  of  delivery, 
method  of  shipment,  methods  of  preservation  and  packaging, 
and  technical  instruction  books  and  data  to  be  delivered. 
The  type  of  contract  desired  (cost  plus  fixed  fee  or  other) 
should  also  be  specified. 

"When  RFP's  are  sent  to  potential  bidders,  the  baBic 
document  will  be  accompanied  by  a  packet  of  associated 
documents  such  as: 

Statement  of  Work 

Technical  Specifications 

Reliability  Specifications 

Special  Instructions 

Each  of  these  documents  will,  in  turn,  reference  military 
standards,  specifications,  exhibits,  and  directives  which 
are  to  become  Integral  to  the  contraot.  For  a  specific 
program,  the  RFP  packets  sent  to  potential  bidders  must 
be  identical  to  assure  an  equal  opportunity  to  each  bidder. 
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2. 2. 2. 4. 2  Reliability  Requirements 
In  Request  for  Proposal 


The  SPO  Reliability  Coordinator  la  obligated  (see 
Table  2-10)  to  assure  that  system  reliability  requirements 
(both  quantitative  and  qualitative)  are  correctly  and 
completely  stated  In  the  appropriate  RFP  documents.  Other 
reliability  considerations  to  be  covered  In  the  RFP  Include: 

(1)  Organization  and  preparation  of  a  reliability 
program  plan.  When  a  bidder  submits  a  proposal, 
he  must  also  submit,  as  a  separate  entity,  his 
plan  for  Implementing  reliability  activities. 

(2)  Distinction  between  the  contents  of  the  desired 
proposal  (see  2.2.3)  and  the  desired  reliability 
program  plan  (see  2.3.2). 

(3)  Reliability  program  activities  requiring  special 
emphasis . 

(4)  Format  of  reports,  program  plans,  and  data 
collection. 

(5)  Encouragement  of  contractor  initiative  in 
suggesting  state-of-the-art  advances  relating 
to  reliability  technology, 


2. 2. 2. 4. 3  Bidders '  Conferences 


In  the  procurement  of  large  or  complex  systems.  It 
Is  often  advantageous  to  both  the  Air  Force  and  industry 
for  the  former  to  hold  bidders'  conferences.  These  con¬ 
ferences  are  attended  by  representatives  from  the  several 
contracting  firms  which  are  considered  to  have  the  capa¬ 
bility  to  produce  the  desired  system.  They  provide  an 
opportunity  for  the  Air  Force  to  give  teohnloal  details 
and  other  special  Information  relating  to  the  system  and 
for  the  bidders  to  ask  questions  which  will  assist  them 
In  preparing  proposals. 

Bidders'  conferences  beoome  almost  essential  In  the 
procurement  of  systems  and  servloes  of  a  classified  nature. 
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2. 2. 2. 5  Problems  and  Special  Considerations 
Relating  to  Preparation  of 
Request  for  Proposal 


2 . 2 . 2 . 5 . 1  Inaccurate  Determination 

of  System  Reliability  Requirements 


Inaccuracy  In  the  determination  of  the  system  relia¬ 
bility  requirements  will,  of  course,  be  reflected  In  the 
Request  for  Proposal  because  that  document  must  describe 
the  qualities  of  the  desired  product.  If  subsequent 
study  indicates  that  a  higher  level  of  reliability  is 
required  than  that  stated  in  the  RPP,  it  can  be  implemented 
in  the  program  only  as  an  increase  in  the  scope  of  work. 

If  It  Is  determined  that  original  reliability  requirements 
were  higher  then  necessary,  the  specifications  can  be 
relaxed  through  contract  negotiation  (see  2.2.4),  but  such 
relaxation  of  requirements  will  not  necessarily  result  in 
a  monetary  saving.  Downgrading  of  specifications  can, 
however,  sometimes  result  in  speeding  up  of  schedules. 


2. 2. 2. 5. 2  Insufficient  Information 
on  Potential  Bidders 


Each  procuring  activity  should  take  special  care  to 
maintain  an  extensive  and  accurate  list  of  qualified 
bidders  (see  2.2.1).  If  insufficient  information  concern¬ 
ing  potential  bidders  exists  when  RFP's  are  to  be  distri¬ 
buted  for  a  particular  program,  time  which  could  be  saved 
by  going  to  specific  bidders  with  the  desired  capability 
is  lost  while  the  procuring  activity  seeks  more  information 
on  bidders.  Further,  RFP's  may  be  sent  to  the  wrong  bidders; 
some  bidders  who  are  qualified  may  be  overlooked  while  other 
unqualified  bidders  are  solicited. 


2. 2. 2. 5. 3  Insufficient  Time  Allowed 
for  Proposal  Preparation 


RFP's  state  a  specific  date  on  which  bidders'  proposals 
are  due  at  the  procuring  activity's  office.  If  bidders  are 
allowed  Insufficient  proposal  preparation  time  between  reoelpt 
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of  the  RFP  and  the  due  date,  they  may  make  hurried  judg¬ 
ments  concerning  the  program.  While  a  bidder's  commitment 
is  his  own  responsibility,  any  misjudgment  in  the  area  of 
reliability  may  eventually  be  reflected  in  failure  of  the 
program.  Sufficient  time  should  be  allowed  for  each 
bidder  to  analyze  the  system  and  its  requirements  thoroughly 
so  that  competent  and  realistic  proposals  will  result. 


2.2.3 


Evaluation  of  Proposals 


2.2. 3.1  Definition 


Evaluation  of  proposals  Is  the  process  of  determining 
which  proposed  program  (and  thus  which  bidder)  Is  most 
suited  to  provide  a  product  or  service  needed  by  the  Air 
Force.  The  evaluation  considerations  discussed  In  2.2.3 
are  those  pertaining  primarily  to  the  bidder's  proposed 
over-all  approach  and  his  qualifications  for  meeting 
reliability  requirements  associated  with  the  Air  Force 
need. 


The  bidder's  specific  reliability  program  plan, 
detailing  procedures,  schedules,  and  assignments  for 
implementing  reliability  activities,  is  to  be  submitted 
as  a  separate  entity  with  his  proposal.  Because 
reliability  program  planning  is  an  activity  which  extends 
beyond  the  proposal  state,  evaluation  of  this  specific 
plan  is  discussed  separately  in  2.3.2. 


2. 2. 3. 2  Air  Force  Policy 


Air  Force  policy  relating  to  evaluation  of  reliability 
considerations  in  proposals  is  Btated  in  Table  2-11. 


2.2. 3. 3  Responsibilities  for  Evaluation 
of  Proposals 


As  shown  in  Table  2-12,  basic  responsibility  for  obtain¬ 
ing  adequate  reliability  commitments  from  contractors  rests 
with  the  SPO  Reliability  Coordinator.  Consequently,  he  la 
responsible  for  analyzing  and  evaluating  the  reliability 
aspects  of  a  bidder's  proposal.  Implicit  to  the  respon* 
sibilityfor  proposal  evaluation  is  knowledge  of  the  type 
and  scope  of  effort  which  any  oontraotor  would  have  to 
put  forth  to  meet  the  reliability  requirements  of  the 
program. 


TABLE  2-11 


SUMMARY  OP  AIR  FORCE  POLICY 
RELATINO  TO  RELIABILITY  CONSIDERATIONS 
IN  PROPOSAL  EVALUATION 


Air  Force 
Document 

Statement  of  Policy 

Par. 

No. 

AFR  80-5 

(l)  Proposals  must  contain  a  description  of 

3-g 

4  June  1962 

the  contractor's  reliability  program  and 
design  strategy  In  sufficient  detail  for 
evaluation  and  Air  Force  decision  during 

design  selection  phases  regarding  adequacy, 
kind,  and  level  of  effort. 

(2)  Contractor 's  reliability  capability, 
considering  both  past  performance  and  pro¬ 
posed  programs,  will  be  a  major  factor  In 
all  source  selection  action. 

4.e 

3 
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TABLE  2-12 

SUMMARY  OP  RESPONSIBILITIES 
RELATING  TO  RELIABILITY  CONSIDERATIONS 
IN  PROPOSAL  EVALUATION 


Air  Force 
Document 

Level 
to  Which 
Assigned 

Statement  of  Responsibility 

Par. 

No. 

APR  80-5 

4  June  1962 

APSC 

(1)  Insure  that  quantitative 
reliability  requirements  are 
specified  in  proposals  or  bids 
requested  from  industry. 

6.b 

(2)  Evaluate  contractor  relia¬ 
bility  program  proposals  in  all 
system  source  selection  actions. 

6.m 

AFBSDR  80-5 
28  Dec.  1962 

SPO 

Reliability 

Coordinators 

Evaluate  all  aspects  of  the  con¬ 
tractor's  reliability  program 
and  take  action  as  necessary  to 
assure  the  adequacy  of  this 
program 

6-b- (3) 

SE/TD 

Contractors 

Review  the  contractor's  relia¬ 
bility  program  submitted  in 
accordance  with  MIL-R-275^2  or 
other  contractual  reliability 
documents . 

6.c.(2) 

2.2-17 


2.2. 3.4  Implementation  of  Evaluation 
of  Proposals 

2. 2.3.4. 1  Evaluation  of  Bidders' 

Reliability  Capabilities 


There  are  three  major  considerations  relating  to 
reliability  which  should  be  covered  In  a  bidder's  proposal 
and  thus  evaluated  by  the  SPO  Reliability  Coordinator: 


(1)  Analysis  of  requirements  —  The  bidder  should 
indicate  recognition  of  and  appreciation  for 
the  nature  of  the  reliability  problems  in¬ 
herent  in  the  program  and  propose  approaches 
for  solution  of  these  problems. 

(2)  Proposed  reliability  program  —  The  bidder 
should  discuss  the  type  and  scope  of  effort 
he  proposes  to  undertake  to  meet  program 
reliability  requirements.  (This  discussion 
of  proposed  effort  is  distinct  from  the 
separate  and  more  detailed  reliability  prog¬ 
ram  plan  which  the  bidder  must  also  submit — 
see  2.3). 

(3)  Qualifications  of  bidder  —  The  bidder  should 
describe  his  capabilities  for  implementing  an 
effective  reliability  effort,  including  quali¬ 
fied  personnel  and  over-all  reliability  organiz¬ 
ation. 


Standard  check  lists  and  evaluation  forms  are  useful 
in  grading  both  proposals  and  bidders  in  procurement  of 
products  and  services,  but  many  of  those  in  use  (e.g., 

APSC  51  and  51A)  do  not  provide  for  evaluation  of  relia¬ 
bility  capability.  Figure  2-17  is  a  sample  questionnaire 
primarily  intended  to  evaluate  this  aspect  of  a  bidder's 
proposal,  taking  into  account  the  bidder's  adequacy  in 
the  three  areas  listed  above  —  analysis  of  the  reliability 
problems,  elements  of  proposed  reliability  program,  and 
qualifications  for  implementing  such  a  program. 


FIGURE  2-  17 

SAMPLE  QUESTIONNAIRE  FOR  EVALUATING  PROPOSALS 


PROPOSAL  RELIABILITY  EVALUATION  QUESTIONNAIRE  (Page  l) 

Company _  RFP  No. 

Address  Date  _ 


I.  ANALYSIS  OF  RELIABILITY  REQUIREMENTS: 

(1)  Does  the  proposal  Include  a  preliminary  analysis  of 
reliability  requirements? 

(2)  Does  the  proposal  demonstrate  correct  interpretation 
of  requirements? 

(3)  Does  the  proposal  set  reliability  in  proper  perspective? 
That  is,  does  the  potential  contractor  show  awareness  of 
the  relationships  between  reliability,  other  system  para¬ 
meters,  and  the  effectiveness  of  the  system? 

(4)  Does  the  proposal  recognize  the  implications  of  the  relia¬ 
bility  requirements?  Does  the  bidder  translate  his  expe¬ 
rience  and  knowledge  into  logical  conclusions  regarding 
scope  of  work  such  that  confidence  may  be  reasonably 
placed  in  initial  estimates  of  probability  of  success 

and  cost?  As  for  example: 

(a)  Will  state-of-the-art  constraints  on  part  life 
preclude  the  achievement  of  requirements  by  use 
of  available  parts? 

(b)  Will  part  improvement  be  necessary  and  feasible? 

(c)  Will  redundancy  be  neceBBary? 

(d)  Do  significant  areas  of  uncertainty  exist  which 
may  require  special  study  and  influence  both 
confidence  and  expected  cost? 

(e)  Does  the  proposal  describe  the  basis  on  which 
Judgments  regarding  the  above  are  made? 

II.  RELIABILITY  PROGRAM  ELEMENTS: 

A.  System  Analysis 

(1)  Does  the  proposed  program  make  adequate  provision  for 
analysis  of  system  requirements  to  better  define  optimum 
configuration  and  equipment  design  criteria?  Will  re¬ 
sults  be  available  on  a  timely  basis? 

(2)  Are  the  influences  of  environment  identified  and  pro¬ 
perly  considered? 
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FIGURE  2-17  (Continued) 

SAMPLE  QUESTIONNAIRE  FOR  EVALUATING  PROPOSALS 
PROPOSAL  RELIABILITY  EVALUATION  QUESTIONNAIRE  (Page  2) 

Company _ RFP  No. _ ___ 

Address  _ Date  _ • _ _ 


II.  RELIABILITY  PROGRAM  EIEMENTS  (Continued): 

A.  System  Analysis  (Continued) 

(3)  Does  the  proposed  program  provide  for  translation  of 
requirements  and  the  results  of  system  analysis  Into 
design  criteria  such  as: 

(a)  Allocation  of  reliability  requirements  to 
equipment  or  black  box  level? 

(b)  Description  of  relative  Importance  of  Indi¬ 
vidual  equipment  to  system  function? 

(c)  Potential  trade-offs  between  reliability 
and  other  system  parameters  to  achieve  given 
effectiveness  for  minimum  resource  cost? 

B.  Part  Selection  and  Application 

(1)  Does  the  pz*oposal  show  availability  of  parts  life  data 
as  necessary  for  satisfactory  part  selection  and  appli¬ 
cation  (includes  derating)? 

(2)  Does  adequate  quality  control  exist  to  Insure  acceptable 
Initial  and  continued  part  quality?  Is  vendor  surveil¬ 
lance  used  effectively  toward  this  end? 

C.  Circuit  Tolerance 

(1)  Does  the  proposal  recognize  the  need  for  attention  to 
circuit  tolerance  to  part  parameter  variations  In  both 
the  population  and  time  domains? 

(2)  Does  the  proposal  show  knowledge  of  techniques  for 
assessing  the  cumulative  effeots  of  part  parameter 
variations  and  for  design  of  clroults  of  optimum 
tolerance? 

D.  Audit  and  Control 

(1)  Does  the  proposed  program  provide  adequate  means  for 

audit  of  progress  to  includes 

(a)  Design  review? 

(b)  Reliability  prediction  and  analysis? 

(c)  Tests  as  appropriate? 

(d)  Special  studies? 


FIGURE  2-  17  (Continued) 


SAMPLE  QUESTIONNAIRE  FOP  EVALUATING  PROPOSALS 


PROPOSAL  RELIABILITY  EVALUATION  QUESTIONNAIRE  (Page  3) 

Company _ _ RFP  No. 

Address  _ _ Date _ 


II.  RELIABILITY  PROGRAM  ELEMENTS  (Continued): 

E.  Verification 

(1)  Does  the  proposal  provide  an  adequate  plan  for  verifi¬ 
cation  of  achieved  reliability? 

(2)  Is  the  operational  environment  properly  accounted  for? 


III.  QUALIFICATIONS  OF  BIDDER: 

7  A .  Qualified  Personnel 

(1)  Does  the  proposal  identify,  by  qualifications,  the  per¬ 
sonnel  to  be  assigned  to  the  reliability  program? 

(2)  Is  the  number  of  personnel,  particularly  of  those  to  be 
assigned  full  time,  adequate  In  the  light  of  the  scope 
of  the  over-all  program? 

(3)  Is  the  experience  of  the  personnel  adequate? 

(4)  Does  the  group  represent  a  good  cross-section  of  quali¬ 
fications  in  important  areas  such  as  analysis,  design, 
statistical  methods,  parts,  and  test  methods? 


B.  Reliability  Organisation 

(1)  Is  the  framework  of  the  bidder's  reliability 

that  the  work  of  the  group  will  be  group  such 

sive  to  over-all  requirementa^^ei^^y*  truly  respon- 
areaa,  and  iffirtl —  \m^ to  problem 

and  the  oont-^^pf^STtne  formulation  of  design  orlterla 
of  design  for  reliability? 

00  the  group  have  adequate  stature  and  authority? 

(3)  Do  means  exist  for  audit  of  progress? 

(4)  Do  means  exist  for  feed  baok  and  exchange  of  Information? 


IV.  PROPOSAL  NON- RESPONSIVE  TO: 


V.  EXCEPTIONS  TAKEN  TO: 


2.2-21 


Numerical  rating  values  could  be  assigned  to  the 
various  elements  of  a  questionnaire  such  as  that  shown 
in  Figure  2-17  if  a  quantitative  assessment  of  proposals 
is  desired.  Weighting  factors  could  aleo  be  assigned 
to  the  questionnaire  sections  to  indicate  the  relative 
importance  of  various  areas  of  bidder  capability  to  the 
requirements  of  a  specific  program.  However,  the  same 
numerical  rating  or  weighting  basis  should  be  applied  to 
the  proposals  of  all  bidders  on  a  particular  procures ent. 


Numerical  rating  features  can  be  used  in  bid 
evaluations  for  procurements  conducted  on  a  "negotiated"* 
basis  but  not  for  procurements  conducted  on  an  "adverti^.* 
basis.  In  "advertised"  procurement,  vendor  eva^r  ~‘*®a 
must  be  solely  on  the  basis  of  responsive 
sive  to  the  request  for  bid. 


^acion 
’not  respon- 


2. 2. 3.4.2 


of  cost 


Cost  should  not  be  a  factor  in  the  initial  technical 
evaluation  of  proposed  reliability  programs.  Estimated 
costs  for  performing  a  proposed  program  should  therefore 
be  presented  in  a  separable  portion  of  bidders'  proposals. 
After  those  proposals  which  offer  adequate  reliability 
effort  have  been  determined,  the  relative  cost  considera¬ 
tions  should  be  reviewed  and  evaluated. 


2.2. 3. 5  Problems  and  Special  Considerations 
Relating  to  Evaluation  of  Proposals 


2.2. 3.5.1  Inadequvts  guidance  In  Request  for  Proposal 


Faulty  or  Inadequate  information  in  an  RFP  may  lead 
to  proposals  which  do  not  meet  the  true  need  of  the  pro¬ 
curement.  Some  oontraotors  may  recognise  the  inadequacies 
of  the  RFP,  reconstruct  the  requirements,  and  submit 
acceptable  proposals.  Others,  however,  may  respond  with 
a  proposal  whloh  answers  the  Inadequate  RFP  but  not  the 
aotual  requirements  of  the  program. 


♦AFSCP  So-l,  Guide  for  Kvaluaters,  19  August  1962. 


If  it  la  Intended  to  apply  weighting  factors  in 
the  evaluation  of  proposals ,  the  stressing  of  retire¬ 
ments  in  the  RFP  should  correspond  to  the  Intended 
weighting  basis.  This  procedure  will  elicit  responses 
from  bidders  which  are  more  Indicative  of  their  willing¬ 
ness  and  ability  to  comply  with  those  requirements  which 
are  of  particular  importance  in  a  program. 


2. 2. 3.6  Additional  References  on 
Evaluation  of  Proposals 


Supplier  Reliability  Assurance  Programs.  M,  H.  Salts, 

Sixth  National  Symposium  on  Reliability  and  Quality 
Control,  January  I960,  pp.  445-448. 

AIR  70-15)  Procurement  Source  Selection  Procedures,  24  April  1962. 


AFSCR  70- 3A,  Procurement  Source  Selection  Board  Procedures, 
11  September  I9&3- 


2.2.4 


Negotiation  of  Contracts 


2.2.4. 1  Definition 


Negotiation  is  the  activity  in  which  the  customer  and 
a  contractor  determine  and  agree  upon  the  exact  terms 
of  the  contractual  relationship  under  which  the  contractor 
will  furnish  a  desired  product  or  service  to  the  Air 
Force.  The  accuracy  and  completeness  with  which  both 
contractor  and  customer  obligations  are  stated  in  a 
contract  have  considerable  effect  on  the  ultimate  success 
or  failure  of  a  program. 


Negotiation  for  Air  Force  procurement  is  conducted 
by  a  team  of  Air  Force  personnel  assisted  by  representatives 
of  SE/TD  contractors.  The  discussion  in  2.2.4  is  primarily 
concerned  with  the  responsibilities  of  the  System  Program 
Office,  and  thus  the  SPO  Reliability  Coordinator,  in 
assuring  that  reliability  requirements  are  adequately 
covered  in  system  contracts. 


2. 2. 4. 2  Air  Force  Policy 


Air  Force  policy  relating  to  negotiation  of  the 
reliability  portion  of  contracts  is  presented  in  Table  2*13. 


2. 2. 4. 3  Responsibilities  for  Contract  Negotiation 


Responsibilities  explicitly  stated  by  Air  Force 
directives  in  regard  to  incorporation  of  reliability 
requirements  in  contracts  are  given  in  Table  2*14. 


2. 2.4.4  Implementation  of  Contract 
Negotiation 


2.2.4.4.1  Fundamental  Assumptions 


At  the  time  negotiation  of  a  contract  is  to  begln,lt 
is  assumed  that  the  following  background  tasks  will  have 
been  completed  by  the  System  Program  Of  fleet 


TABLE  2-13 

SUMMARY  OP  AIR  FORCE  POLICY 
RELATING  TO  RELIABILITY  CONSIDERATIONS 
IN  CONTRACT  NEGOTIATION 


Air  Force 
Document 

Statement  of  Policy 

Par. 

No. 

APR  80-5 

4  June  1962 

(1)  System  contracts  will  Include  a  require¬ 
ment  for  a  comprehensive  contractor  relia¬ 
bility  program.  Including  quantitative 
requirements  and  tests  for  reliability 
demonstration. 

4.  c 

(2)  If  contract  reliability  requirements 
are  not  met,  or  if  the  contractor's  relia¬ 
bility  effort  is  decreased,  the  decision 
to  accept  or  reject  the  end  item  or  the 
revised  reliability  program  will  be  considered 
with  a  view  toward  monetary  penalties,  unit 
price  decreases,  or  other  considerations 
deemed  equitable. 

4.  f 

AFSCR  80-1 

14  Dec.  1962 

New  programs  will  include  adequate  funds  for 
reliability  effort  in  design  proposals  and 
initial  program  funding.  Specific  activities 
may  be  identified  as  reliability  effort  in 
order  to  provide  a  management  tool  for  guid¬ 
ing  and  assessing  adequacy  of  effort. 

^•g 

AFBSDR  80-5 

28  Dec.  1962 

Adequate  funds  will  be  made  available  in  the 
initial  program  funding  for  ballistic  missile 

By stem  programs  to  provide  for  the  Implementa¬ 
tion  of  a  complete  and  well  balanced  relia¬ 
bility  program. 

-  -  -J 

5-d 

TABLE  2-14 


j  SUMMARY  OP  RESPONSIBILITIES 

1  RELATING  TO  RELIABILITY  CONSIDERATIONS 

* 

IN  CONTRACT  NEGOTIATION 


Air  Force 
Document 

Level 
to  Which 
Assigned 

Statement  of  Responsibility 

Par. 

No. 

AFR  80-5 

4  June  1962 

AFSC 

(l)Assume  responsibility  for  system  pro¬ 
curement  reliability  activities,  including 
quality  control,  production,  packaging, 
transportation,  and  storage. 

6-g 

(2)  Establish  and  fund  basic  and  applied 
research  programs  for  reliability  based  on 
current  and  future  system  requirements. 

6- j 

AFSCR  80-1 

14  Dec.  1 

HQ  AFSC 

stems 

(l)  Prescribe  policies  and  procedures 
necessary  to  assure  contractual  require¬ 
ments  for  reliability  in  each  system, 
subsystem,  or  equipment  being  developed 
for  the  Air  Force  inventory. 

5.a.(3) 

AFSC 

?CS/ 

Procurement 

and 

Materiel 

(1)  Prescribe  procurement  policies  for  in¬ 
clusion  of  reliability  requirements  in 
contracts  and  invoke  monetary  penalties, 
unit  price  decreases,  or  other  considera¬ 
tions  deemed  equitable  if  requirements 
are  not  met. 

5-b. (l) 

System 

Program 

Offices 

(SPO's) 

(l)  Incorporate  reliability  requirements, 
including  provisions  for  demonstration  of 
reliability  in  all  specifications,  exhibits, 
product  descriptions,  or  other  contractual 
documents  to  be  referenced  in  contracts  for 
weapon  and  support  systems  and  associated 
materiel  (Including  government-furnished 
equipment).  Prime  contractors  will  be 
required  to  impose  adequate  requirements 
on  subcontractors  and  vendors.  Contractual 
documents  will  Include  any  specific  require¬ 
ments  related  to  the  control  of  manufactur¬ 
ing  processes  necessary  to  insure  the 
desired  reliability. 

7-c 

(2)  Incorporate  requirements  for  quantita¬ 
tive  data  in  contractual  documents. 

T-e 

2.2-26 


(1)  System  reliability  requirements  will  have  been 
thoroughly  determined  in  the  conceptual  phase 
of  the  program,  and  numerical  values  will  have 
been  established  for  all  reliability  para¬ 
meters  (see  2.1.1  and  2.1.2). 

(2)  The  major  activities  in  which  a  contractor  will 
have  to  engage  to  fulfill  the  program  reliability 
requirements  will  have  been  determined,  and  pre¬ 
liminary  documents  such  as  a  Statement  of  Work 
and  a  Request  for  Proposal  will  have  been 
adequately  prepared  so  as  to  communicate  program 
requirements  to  bidders  (see  2.1.3  and  2,2.2) 

(3)  Proposals  will  have  been  received  and  evaluated 
with  great  care,  and  the  contractor  having 

the  highest  probability  of  success  will  have 
been  selected  (see  2.2. 3) • 


2. 2. 4. 4. 2  Listing  of  Weaknesses 

in  Contractor's  Program 


As  a  result  of  thorough  evaluations  of  the  contractor's 
proposal,  including  his  proposed  reliability  program,  a 
11s  t  of  items  requiring  negotiation  should  have  been  deter¬ 
mined.  These  items  may  be  the  result  of  ambiguity,  omission, 
error,  or  misunderstanding  on  the  part  of  the  contractor 
or  of  pre-contract  changes  in  required  scope  of  work  on 
the  part  of  the  Air  Force.  They  must  be  clarified  in  the 
Statement  of  Work,  the  contractor's  reliability  program 
plan,  and  the  final  contract. 


2. 2. 4. 4. 3  Presentation  of  Air  Force  View 


The  Air  Force  should  present  its  view  regarding  the 
questionable  items  in  the  oontraotor's  proposed  program^ 
and  allow  the  contractor  to  study  eaoh  item  before  replying. 


2. 2. 4. 4. 4  Reply  of  Contraotor 


After  review  of  the  Air  Force  comments  on  his  pro¬ 
posed  reliability  and  other  program  efforts,  the  oontraotor 
may  be  expeoted  to  take  one  of  several  actions  1 


(1)  Accept  the  Air  Force  view  without  exception 
and  make  necessary  adjustments  In  proposed 
program. 

(2)  Accept  the  Air  Force  view  but  propose  cost 
revision  because  of  changes  to  be  made  In  the 
program. 

(3)  Take  exception  to  the  Air  Force  view  on 
technical  grounds. 

(4)  Take  exception  to  the  Air  Force  view  on 
both  technical  and  monetary  grounds. 


2. 2. 4. 4. 5  Areas  Not  to  Be  Negotiated 


There  are  some  considerations  which,  by  Air  Force 
directive,  are  not  subject  to  negotiation  for  a  system 
procurement  contracts 

(1)  Numerical  reliability  requirements  which  are 
acceptable  to  the  Air  Force  must  be  a  part 
of  the  contract. 

(2)  A  reliability  program  plan  which  is  acceptable 
to  the  Air  Force  as  being  adequate  to  meet 

the  system  needs  must  be  a  part  of  the  contract 
(see  2.3). 


(3)  Use  contractor  must  establish  management  pro¬ 
cedures  which  will  assure  control  of  those 
factors  in  research,  development,  manufacture, 
shipment,  and  storage  which  will  result  in  the 
required  level  of  product  reliability. 

(4)  The  contract  must  contain  penalty  clauses  for 
non-fulfillment  of  contractual  reliability 
requirements. 


2. 2. 4. 4. 6  Control  of  Agenda 

In  Contract  Negotiation 


Contract  negotiation  should  be  done  in  formal  meetings 
between  an  Air  Force  negotiating  team  and  the  contractor's 
delegated  representatives.  The  Air  Force  should  maintain 
control  of  these  meeting  through  the  following  actions* 

(1)  Assigning  a  member  of  the  Air  Force  team 
as  chairman  of  the  meeting  so  that  order 
and  progress  can  be  maintained. 

(2)  Pre-delivering  the  list  of  negotiable  points 
and  a  proposed  meeting  agenda  to  the  contractor. 

(3)  Establishing  a  schedule  for  the  agenda  bo  that 
the  meeting  time  will  be  efficiently  used. 


2.2.4. 5  Problems  and  Speciax  Considerations 
Relating  to  Contract  Negotiations 

2.2.4. 5.1  Level  of  Contract  Negotiation 


Different  contractors  place  the  responsibility  for 
contract  negotiation  at  various  levels  within  their  company 
structures.  The  Air  Force  should  endeavor  to  negotiate 
a  system  contract  with  high  officials  of  the  company  in¬ 
volved  in  order  to  assure  management  recognition  of  con¬ 
tractual  objectives  and  obligations.  However,  in  nego¬ 
tiation  of  the  reliability  phases  of  a  contraot,  the  Air 
Force  should  insist  that  key  personnel  of  the  contractor 
reliability  organization  be  present  to  assure  that  cor¬ 
porate  commitments  are  in  keeping  with  the  actual  capa¬ 
bilities  of  the  reliability  organization. 


2. 2. 4. 5. 2  Amount  of  Negotiation  Required 


If  the  Air  Force  Statement  of  Work  and  Request  for 
Proposal  were  adequately  prepared  and  the  contractor's 
proposal  was  well  received  and  executed,  there  nay  be  a 
few  points  to  negotiate  in  arriving  at  a  oontraot. 


However,  if  less  than  ideal  conditions  prevail  and 
there  are  many  points  to  negotiate,  the  course  to  be 
followed  will  depend  upon  the  criticalness  of  the  factors 
to  be  negotiated  and  the  time  allowed  for  negotiation. 


If  insufficient  time  is  allowed  for  negotiation, 
both  the  Air  Force  and  the  contractor  are  prone  to  make 
concessions  deleterious  to  the  success  of  the  over-all 
program  and  reliability  effort.  By  controlling  the 
agenda,  listing  the  negotiable  points,  and  screening 
them  for  relative  importance  prior  to  actual  negotiation 
meetings,  the  Air  Force  negotiation  team  chairman  should 
be  able  to  allow  sufficient  negotiation  time  and  to 
accomplish  his  goals  in  the  allowed  time. 


2. 2. 4. 5. 3  Agreement  on  Program  Details 


Agreement  on  general  principles  involved  in  a  system 
program  should  have  been  reached  before  the  contract 
negotiation  stage  is  reached,  through  the  activities  of 
selecting  qualified  bidders  and  evaluating  proposals. 
Detail  requirements  for  the  program  are  the  problems  to 
be  dealt  with  in  contract  negotiation.  In  respect  to 
reliability,  agreement  must  be  attained  on  the  following 
points: 

(1)  What  are  the  requirements  to  be  placed  on 
the  reliability  program? 

(2)  How  will  the  reliability  program  accomplish 
these  requirements? 

(3)  When  will  these  tasks  be  completed? 

(4)  How  will  fulfillment  of  requirements  be 
measured? 

(5)  What  is  the  monetary  value  of  each  program 
activity  and  phase  and  what  penalties  will 

be  evoked  for  non-fulfillment  of  requirements? 


2.3  Enactment  of  Reliability  Program  Plan 


A  third  vital  factor  in  the  establishment  of  a  relia¬ 
bility  program  is  the  enactment  of  an  effective  reliability 
program  plan.  The  importance  of  the  program  plan  is 
emphasized  through  the  realization  that: 

(1)  It  serves  as  the  contractor’s  primary  statement 
of  his  intended  methods  for  implementing  the 
elements  of  the  reliability  program  and  accomp¬ 
lishing  the  requirements  of  the  contract. 

(2)  When  approved,  it  constitutes  an  official  re¬ 
quirement  within  the  over-all  system  program 
and  thus  becomes  a  vital  working  document  and 
primary  source  of  information  for  contractor 
personnel  involved  In  the  reliability  effort. 

(3)  It  serves  the  SPO  Reliability  Coordinator  and 
other  Air  Force  representatives  as  a  tool  for 
measuring  progress  and  determining  fulfillment 
of  contractual  obligations . 

Air  Force  policy  relating  to  enactment  of  a  relia¬ 
bility  program  plan  is  summarized  in  Table  2-15.  The 
assigned  responsibilities  of  SPO  Reliability  Coordinators 
and  other  associated  levels  of  authority  are  presented  in 
Table  2-16.  From  the  tabulated  information.  It  may  be 
concluded  that  the  prime  tasks  of  the  SPO  Reliability  Co¬ 
ordinator  in  enacting  an  effective  reliability  program 
plan  are  to: 

(1)  Provide  the  contractor  with  sufficient  direction 
and  guidance  to  enable  him  to  prepare  the  plan. 

(2)  Review  and  evaluate  the  contractor’s  plan. 

Ideally,  for  a  particular  program,  instructions  would 
be  given  only  once,  and  evaluation  for  acceptance  would 
also  represent  a  single  occurrence.  However,  the  Importance 
of  the  program  plan  and  the  degree  of  detail  with  which  it 
should  be  prepared  suggest  that  it  is  more  practical  for 
enactment  of  the  plan  to  be  accompanied  by  a  period  of 
planned  development.  This  development  would  Involve  repeated 
instruction  and  evaluation  events,  starting  with  the  Initial 
contact  with  prospective  contractors  and  ending  with  the 
offlolal  customer  approval  of  the  program  plan. 
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TABLE  2-15 

SUMMARY  OF  AIR  FORCE  POLICY 
RELATING  TO  ENACTMENT  OF  RELIABILITY  PROGRAM  PLAN 


Air  Force 
Document 

Statement  of  Policy 

Par. 

No. 

AFR  80-5 

4 , June  1962 

(1)  During  the  conceptual  and  acquisition 
phases,  systems  will  be  analyzed  and  a  relia¬ 
bility  program  established  for  each.  Each 
program  will  Include  a  minimum  acceptable 
reliability  level  as  well  as  a  reliability 
goal,  with  Intermediate  quantitative  values 
required  to  measure  progression. 

*.a.(3) 

(2)  System  contracts  will  include  a  require¬ 
ment  for  a  comprehensive  reliability  program. 
Including  quantitative  requirements  and  tests 
for  reliability  demonstration. 

4.  c 

AFSCR  80-1 

14  Dec.  1962 

(l)  Integrated  test  plans  to  investigate 
causes,  effects,  and  modes  of  failure,  and 
to  demonstrate  achieved  reliability  will  be 
developed  and  Incorporated  in  contractual 
documents .  Plans  will  be  designed  to  provide 
the  maximum  information  and  assurance  con¬ 
sistent  with  the  state-of-the-art. 

4.f 

TABLE  2-16 

SUMMARY  OP  RESPONSIBILITIES 


RELATING  TO  ENACTMENT  OP  RELIABILITY  PROGRAM  PLAN 


Air  Force 
Document 

Level 
to  Which 
Assigned 

Statement  of  Responsibility 

■ 

Par. 

No. 

APR  80-5 

4  June  1962 

APSC 

(l)  Incorporate  quantitative  reliability 
requirements  and  provisions  for  demon¬ 
strating  reliability  in  all  specifica¬ 
tions,  exhibits,  product  descriptions, 
work  statements,  and  contractual  clauses 
to  be  referred  to  or  included  In  con¬ 
tracts  for  systems  and  associated  mater¬ 
iel. 

6.c 

(2)  Determine  the  adequacy  of  each  Air 
Force  contractor's  reliability  program 
for  achieving  and  demonstrating  Air 

Force  product  reliability  goals. 

6.d 

(3)  Evaluate  contractor  reliability  pro¬ 
gram  proposals  in  all  system  source 
selection  action. 

6.m 

APSCR  80-1 

14  Dec.  1962 

System 

Program 

Offices 

(SPO's) 

(l)  Establish  the  minimum  amount  of 
detailed  reliability  program  informa¬ 
tion  required  from  contractors  and 
supporting  Air  Force  agencies . 

9- a 

(2)  Define  for  contractor  purposes  an 
adequate  and  comprehensive  contractor 
reliability  program  for  weapon  and  sup¬ 
port  systems,  associated  material,  and 
government- furnished  subsystems . 

7-d 

APBSDR  80-5 
28  Dec.  1962 

SPO 

Reliability 

Coordinators 

(l)  Evaluate  all  aspects  of  the  con¬ 
tractor's  reliability  program  and  take 
action  as  necessary  to  assure  the 
adequacy  of  this  program. 

6.b.(3) 

SE/TD 

Contractors 

(l)  Be  responsible  for  systems  engi¬ 
neering  and  technical  direction  of  the 
associate  contractor's  reliability  pro¬ 
gram,  including  detailed  instructions 
regarding  implementation  of  contrac¬ 
tually  referenced  reliability  documents, 
to  the  extent  established  by  the  tech¬ 
nical  directorate  concerned. 

6.c.(i) 

(2)  Review  the  contractor's  reliability 
program  submitted  in  accordance  with 
MIL-R-27542  or  other  contractual  relia¬ 
bility  documents. 

6.c. (2) 

(3)  Evaluate  and  assure  that  data  and 
assumptions  used  by  the  contractor  are 
valid  prior  to  preparation  of  plana  for 
demonstration  of  achieved  reliability. 

6.c.(3) 
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It  is  entirely  possible  for  the  SPO  Reliability  Coordinator 
to  engage  in  a  sequence  of  any  or  all  of  the  following 
typical  actions  (listed  in  chronological  order)  relating 
to  enactment  of  the  reliability  program  plan: 

(1)  Within  the  Request  for  Proposal,  direct 
potential  bidders  to  submit  a  reliability 
program  plan  as  part  of  their  proposal. 

(This  action  is  required  by  MIL-R-27542A.) 

(2)  Issue  supplemental  instructions  on  specific 
requirements  for  the  plan. 

(3)  Evaluate  bidders'  program  plans  as  submitted 
with  their  proposals. 

(4)  Shortly  after  contract  award,  conduct  a 
briefing  for  the  selected  contractor  to 
clarify  and  interpret  requirements  for 
the  reliability  program  plan  and  issue 
additional  instructions. 

(5)  Evaluate  the  contractor's  plan  as  submitted 
subsequent  to  the  post-award  briefing. 

(6)  Provide  comments  indicating  necessary  con¬ 
tractor  actions  in  order  for  the  plan  to  be 
acceptable . 

(7)  Re-evaluate  the  contractor's  final  draft  and 
approve  the  plan  if  acceptable,  (if  the  plan 
is  not  acceptable,  events  6  and  7  should  be 
repeated  until  acceptance  is  attained.) 

The  above  procedure  may  seem  to  represent  an  administrative 
burden  on  both  the  SPO  Reliability  Coordinator  and  the  con¬ 
tractor,  but  past  experience  has  shown  that  when  comprehen¬ 
sive  programs  are  involved,  this  degree  of  customer- contractor 
coordination  is  neoessary. 

The  time  intervals  between  instruction  and  evaluation 
events  are  dependent  on  the  circumstances  of  individual 
programs.  One  typical  sequence  of  events  which  can  be  used 
as  guidance  was  set  forth  in  the  General  Work  Specifications 
for  Atlas  Standard  Spaoe  Launch  Vehicle  Reliability  Assur¬ 
ance  and  Quality  Assuranoe  Requirements  (Aerospaoe  Corpora¬ 
tion  Dooument  No.  1923 . 13^7 »  dated  August  1962;: 


(1)  A  briefing  meeting  was  scheduled  within  15  days 
after  contract  start  to  instruct  the  contractor 
as  to  the  requirements  of  the  work  statement 
and  their  relation  to  the  reliability  program 
plan. 

(2)  The  contractor  was  required  to  submit  a 
preliminary  draft  of  his  program  plan  30  days 
after  the  briefing  meeting. 

(3)  The  customer  (in  this  case,  SSD)  was  to  review 
the  draft  with  the  contractor  and  submit 
detailed  comments  to  him  within  15  days  after 
receipt  of  the  draft. 

(4)  The  contractor  was  required  to  submit  a  final 
draft  for  customer  approval  within  30  days 
after  receipt  of  comments  on  the  preliminary 
draft . 
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2.3.1  Instructions  to  Contractor 


2. 3. 1.1  Definition 


This  activity  Includes  preparation  of  any  statement 
of  work  (see  2.1.3),  supplemental  written  Instruction, 
post-award  briefing,  or  other  guidance  material  which  Is 
Intended  to  direct  the  contractor  In  preparation  of  his 
reliability  program  plan.  Such  action  by  the  SPO  Relia¬ 
bility  Coordinator  normally  commences  at  the  time  the 
RFP  is  prepared  and  distributed  and  continues  until 
shortly  after  contract  award. 


2 . 3 • 1 • 2  Air  Force  Policy 


Air  Force  policy  relating  to  preparation  of  speci¬ 
fications  or  instructions  for  a  reliability  program  plan 
Is  covered  by  the  material  in  Table  2-15  (see  2.3). 


2. 3. 1.3  Responsibilities  for  Contractor 
Instructions 


The  responsibilities  of  SPO  Reliability  Coordinators 
relating  to  preparation  of  specifications  or  instructions 
for  a  reliability  program  plan  are  indicated  in  Table  2-16 
(see  2.3) . 


2.3.1.^  Implementation  of  Contractor 
Instructions 


It  would  be  desirable,  of  course,  if  all  instructions 
pertinent  to  preparation  of  a  reliability  program  plan 
could  be  made  available  to  potential  bidders  within  the 
RFP.  However,  the  time  constraints  which  normally  prevail 
and  the  existence  of  unknown  factors  vital  to  a  program 
usually  necessitate  that  the  instruction  activity  be  a 
continuing  effort,  extending  past  the  proposal  phase  into 
an  initial  period  after  contract  award. 
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The  detailed  instructions  provided  to  the  selected 
contractor  should  represent  an  interpretation  or  ampli¬ 
fication  of  original  statements  appearing  within  the  basic 
contract  document.  These  instructions  should  be  presented 
to  the  contractor  as  early  as  practicable,  through  formal 
written  directives  or  informal  briefings  (either  presenta¬ 
tion  method  can  be  equally  effective). 

The  following  factors  relating  to  the  reliability 
program  plan  should  be  the  subject  of  the  special  instruc¬ 
tions  to  the  contractor: 

(1)  Schedule  for  submission  of  plan  —  A  precise 
statement  which  indicates  the  number  of  pre¬ 
liminary  submittals  and  brief ingB,  and  the 
dates  for  these  events  and  the  final  submittal. 

(2)  Format  of  plan  --  A  statement  of  the  mechanical 
requirements  for  submission  of  the  plan,  e.g., 
size  of  paper  to  be  used,  copies  to  be  bound  or 
unbound,  reproducible  copy  to  be  submitted,  etc. 

(3)  Outline  of  plan  —  A  brief  listing  of  the  sub¬ 
jects  to  be  covered  in  the  plan.  For  example, 
the  major  sections  could  be: 

(a)  Scope 

(b)  Applicable  documents 
(o)  Elements  of  the  program 

1 .  Management  elements 

2 .  Technical  elements 

(d)  Schedules  for  the  accomplishment  of  work 

(e)  Allocations  of  manpower 

(4)  Identification  of  elements  of  program  —  A  state¬ 
ment  whloh  describes  the  management  and  technical 
elements  to  be  Incorporated  in  the  program.  An 
outline  of  the  statement  of  requirements  appearing 
In  the  baslo  contract  dooument  may  serve  this 
purpose. 
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(5)  Categories  of  information  for  each  program 
element  —  A  statement  of  the  general  information 
desired  for  each  task  of  the  program,  including: 

(a)  Assignment  of  task  responsibility  to  a 
specific  working  group  within  the  con¬ 
tractor's  organization. 

(b)  Procedure  to  be  employed  in  implementing 
the  task  (description  or  reference  to 
appropriate  procedural  document) . 

(c)  Schedule  for  accomplishment  of  signif¬ 
icant  events  associated  with  the  task 
(or  identification  of  the  times  at  which 
future  planning  decisions  will  be  made). 

(6)  Instructions  related  to  specific  elements  of 
work  --  Special  Instructions  related,  in  gen¬ 
eral,  to  procedures  and  schedules  for  specific 
program  tasks. 

Table  2-17  illustrates  the  degree  of  detail  with  which 
the  SPO  Reliability  Coordinator  should  provide  guidance  or 
instructions  to  the  contractor.  The  list  of  subjects  (ele¬ 
ments  of  work)  in  the  table  is  not  exhaustive  but  does  re¬ 
present  major  areas  of  activity  common  to  most  programs. 
Instructions  appropriate  to  a  particular  program  could  be 
tabulated  by  reviewing  the  contractual  statement  of  work 
for  the  program.  These  prepared  instructions  should  also 
be  used  later  as  a  basis  for  evaluation  of  the  reliability 
program  plan  submitted  by  the  contractor  (see  2.3.2). 


D 


2. 3. 1.5  Problems  and  Special  Considerations 
Relating  to  Contractor  Instructions 


2. 3. 1.5.1  Influence  of  System  Development 

Phase  on  Complexity  of  Program  Plan 


A  reliability  program  plan  will  become  more  complex 
as  the  system  development  progresses.  The  program  plan 
for  the  feasibility  phase  of  a  system  program  will  be  some¬ 
what  theoretical  in  approach  and  emphasize  potential  relia¬ 
bility  relationships.  As  the  program  prooeeds  to  other 
phases  —  such  as  exploratory,  advanced,  and,  finally, 
operational  development  —  reliability  requirements  will 
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TAM  2-17 

TYPICAL  INSTRUCTIONS  TO  CONTRACTOR 
POP  PRIPARINO  PXU ABILITY  FROORAH  PLAN 


Subject 

Instructions  Pels tin#  to  Procedure 

Instructions  Reis ting  to  Sohedule 

Profraa  Nanaftaent 

(1)  Present  an  organ! set 1 on* 1  chert  for  tho  oompeny 
which  Illustrates  ths  corporate  structure  for 
menegemtnt  and  lndlosts  ths  spsolfle  blooks  whloh 
will  Maintain  responsibilities  for  Implementing 
tasks  under  the  reliability  progrem. 

(2)  Describe  the  relationship  between  any  oentral 
reliability  croup  and  that  group  whloh  la  a  part 
of  the  organisation  for  this  particular  progrem. 

(3)  Indicate  the  nuaber  of  personnel  (aotual  and 
planned)  who  will  staff  eaeh  working  group  whloh 
will  perform  work  under  this  reliability  prograa. 

(4)  Reference  or  incorpora  e  those  top  management 
policy  directives  which  Institute  reliability 
provisions  on  a  ooapany-wlde  basis. 

(5)  Reference  or  Incorporate  those  prograa  manage-  . 
ment  directives  whloh  are  specifically  generated 

In  support  of  this  program. 

If  ra-oraanlaatlon  la  currently  planned,  ahow 
tha  point  In  tlno  at  whlah  thla  ra-orfanlaatlon 
will  occur. 

Indlaata  planned  growth  on  a  oaltndar  tint  boat. 

If  thaaa  dlraotlvao  are  currently  not  .In  exlat- 
anoo.  Indicate  tha  data  by  whloh  they  will  ba 
eatabllehed. 

If  thaaa  dlraetlvaa  are  aurrontly  aohedultd  far 
future  publlaatlon,  Indloato  tho  oahodulod  date. 

Prediction  Techniques 

(1)  Describe  the  prediction  technique (s)  to  be  used 
in  this  prograa  and  relate  these  to  anticipated 
information  availability.  If  a  standard  method (s) 
exists  within  the  company ,  Incorporate  this 
standard  Into  the  prograa  plan. 

(2)  Indicate  the  specific  manner  by  whloh  Information 
on  predictions  will  flow  between  appropriate 
working  groups. 

(3)  Indicate  ths  number  and  frequency  of  reliability 
predlctlone  which  will  be  made  In  support  of  the 
program. 

(4)  indicate  the  method  of  submitting  the  reports  of 
predictions  made. 

(5)  Describe  the  Intended  method  of  acquiring  failure 
rate  Information  for  use  In  reliability  predic¬ 
tion. 

If  a  standard  manual  for  performing  predictions 
is  to  bo  prepared  in  support  of  tho  program, 
Indloato  the  scheduled  date  of  publication. 

Show  the  schedule  for  performing  predictions, 
and  give  the  descriptive  tit lee  by  which  these 
predictions  are  Identified. 

Show  the.  time  schedule  for  submittal  of 
prediction  roporta. 

Document  Rsris* 

(1)  List  the  categories  of  documents  whloh  will  be 
subjected  to  review  by  representatives  of  the 
reliability  organisation. 

*"  ir.'lcate  whloh  of  the  doouaents  In  (1)  will  be 
signed-off  by  reliability  personnel. 

(3)  Trdloate  how  reviewers  will  be  considered  as 
qualified  to  perform  the  function  of  doouaent 
review. 

(4)  indicate  what  standards  will  be  used  by  reviewers 
in  reviewing  documents.  If  a  company  standard 
currently  exists,  Incorporate  this  document  Into 
the  reliability  program  plan. 

(3)  Indicate  the  manner  In  which  discrepancies  noted 
during  doouaent  review  will  be  Incorporated  into 
the  document. 

Indloato  tho  date  the  document  rovlow  activity 
wi"l  owmenoe . 

If  a  standard  la  to  be  generated,  Indloato  the 
Intended  date  of  laauanoo. 

Design  Review 

(1)  Identify  th*  types  of  daalfn  r*Yim  to  0.  oon- 
duotad.  («.«.,  ijitn,  lubayatam,  and  ooaponant.) 

(2)  Tltl.  and  dafin*  by  rofor.no.  to  .tat.  of  dotlfn 
maturity  th.  d.llgn  rovlooa  to  bo  aado  for  oaoh 
ayaton  lovol  (e.f .,  prellalnary,  dotall.d ,  and 
final). 

(3)  Indlaata  th.  personnel  who  will  partlalpato  In 
design  rovlowa. 

(A)  Indlaata  who  will  aorvo  a.  chairman  for  daalfn 
rovlowa . 

(3)  Indlaata  aannar  In  whlah  approprlato  Air  Vera, 
poraonnol  will  b«  advlaod  aa  to  farthaaalng 
daalfn  rovlowa. 

(6)  Daaarlba  tha  aannor  In  whlah  noaaaaary  propara- 
tlona  for  doalfn  rovlowa  (a.g,,  preparation  of 
af.nda  or  dtalnatlon  of  loaatien  for  rovlow) 
will  bo  aaaaapllshtd . 

(T)  In corpora  to  or  rofar  to  any  ahaak  11a to  whlah 
will  aa  uiad  in  tha  aanduat  ef  d.alcn  ravltwa. 

(8)  Inaorporata  or  rafaranaa  any  aaapany  or  profran 
proaadural  atandarda  whlah  will  govam  tha  o on- 
duct  of  daalfn  ravltwa. 

(9)  Daaarlba  tha  aannar  In  whlah  daalfn  ravlaw  pro- 
atodlnft  will  ba  raaordad  and  rapartad. 

(10)  Daaarlba  tha  aannar  In  whlah  dlaereaanelee  no  tod 

freeent  a  aahedula,  baaad  an  aalandar  tine, 
whloh  indlaata*  whan  aaoh  of  thaaa  will  ba 
oonduotad.  If  aufflolant  information  la  not 
available  at  tlaa  of  prograa  plan  oubalttal, 
Indlaata  aohadula  Information  ta  ttia  extant 
poaalbla  and  daaarlba  how  edbgeguent  eehadullng 
will  bo  davolopod  and  aubalttad  ta  tha  Air  Perce . 

If  thee,  ahaak  lleta  are  ta  ba  prepared  at  a 
future  data,  Indlaata  tlaa  of  ■ahodulad 
publlaatlon. 

If  thaaa  atandarda  are  to  ba  prepared  at  a 
future  data,  Indlaata  tlaa  of  oahodulod 
publlaatlon. 

Indlaata  tlaa  Interval  between  dailfr  rovlow 
aanduat  and  laauanoo  of  the  daalfn  ravlaw  report. 

Demons  trmUsm  of 

Achieved  MUtMUty 

(1)  Describe  tha  general  plan  prepot td  far  ianonetre- 
tlnf  reliability  aahleveaant.  Indlaata  baa 
nooaaaary  background  atudlee  will  ta  appraaahod 
and  what  troda-offa  will  bo  oenaldorod  with 
roapoat  to  determining  teat,  tonfldtntt,  laapli 
alaaa.  ata. 

(2)  Indlaata  how  aaauatd  falluro  dlatrlbutlona  will 
ba  vail da tod. 

(3)  Indloato  tho  oyotoa  lovol  ta  ba  aonalfarod  la 
danonatratlan. 

If  tha  ’faaereli  (or  "pral'llatry*)  plan  tm 
daaanatrdtlaa  till  ta  auflttad  iaparatily 
fraa  Me  rallatlUty  peegrea  plan,  ladleate 
tha  data  af  fatal ttal. 

dataUad  «  "ftaalr  lip— trt tlaa  plaa. 
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become  Increasingly  more  specific  and  applicable  to  a 
larger  number  of  proposed  system  equipments.  Consequently, 
reliability  program  plan  requirements  must  become  more 
definitive  and  increase  in  number  and  technical  level. 

Enactment  of  comprehensive  and  complex  program  plans 
may  pose  technical  knowledge  requirements  that  are  dif¬ 
ficult  for  the  SPO  Reliability  Coordinator  to  satisfy 
alone.  Therefore,  consultants  in  specialized  fields 
(such  as  those  provided  by  SE/TD  contractors)  and  review 
and  technical  writing  assistance  should  be  utilized  as 
necessary  to  assure  preparation  of  complete  and  satis¬ 
factory  program  plan  instructions  for  system  contractors. 
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2.3.2 


Evaluation  of  Contractor's 
Program  Plan 


2. 3.2.1  Definition 


The  contractor's  proposed  reliability  program  plan 
must  be  evaluated  by  the  SPO  Reliability  Coordinator 
to  determine  Its  effectiveness  as  a  contractual  and 
management  document  for  assuring  the  achievement  of  the 
desired  reliability  requirements.  The  method  of  per¬ 
forming  this  evaluation  is  not  specified  in  Air  Force 
reliability  directives,  but  it  may  involve  determination 
of  a  rating  (numerical  or  otherwise)  for  the  program 
plan  based  on  comparison  with  a  check  list  or  a  speci¬ 
fication  of  program  plan  requirements.  The  evaluation 
activity  is  usually  considered  to  Include  preparation 
of  a  report  or  comments  noting  any  observed  discrepancies. 


In  evaluating  a  reliability  program  plan,  the  Air  Force 
SPO  Reliability  Coordinator  should  make  full  use  of  the 
technical  and  engineering  consulting  facilities  avail¬ 
able  within  the  appropriate  system  command  (i.e.  command  staffs, 
Aerospace  Corp.,  etc.) 

2. 3. 2. 2  Air  Force  Policy 


Air  Force  policy  relating  to  evaluation  of  a  con¬ 
tractor's  reliability  program  plan  is  covered  by  the 
material  in  Table  2-15  (see  2.3). 


2. 3. 2. 3  Responsibilities  for  Program 
Plan  Evaluation 


The  responsibilities  of  SPO  Reliability  Coordinators 
relating  to  evaluation  of  a  contractor's  reliability  program 
plan  are  Indicated  In  Table  2-16  (see  2.3). 
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t 


Implementation  of  Program 
Plan  Evaluation 


2.3«2.4.1  Fundamental  Bases  of  Evaluation 


In  performing  an  evaluation  of  a  contractor's 
reliability  program  plan,  there  are  several  fundamental 
questions  to  be  answered: 


(l)  Is  the  document  technically  sound?  Or  are 

there  certain  areas,  activities,  or  reliability 
tasks  which  are  technically  unsound  or  require 
unusual  advances  in  the  state-of-the-art? 


(2)  Are  the  managerial  aspects  such  that  adequate 
management  and  administrative  control  of  the 
program  is  assured? 

(3)  Is  the  program  plan  designed  as  a  working 
document? 

(4)  Does  it  provide  sufficient  program  coverage 
and  in  enough  detail  so  that  amendments  will 
be  kept  to  a  minimum? 

(3)  Does  it  meet  or  exceed  the  instructions  and 

directives  which  were  issued  to  the  contractor? 


2,3*2.4.2  Performance  of  Spec if lo 
Evaluations 


In  evaluation  of  a  reliability  program  plan,  it  is 
Impractical  to  take  a  purely  qualitative  approach  and 
make  a  single  Judgment  as  to  the  adequacy  or  Inadequacy 
of  the  document.  Such  an  approach,  while  simple  to 
administer,  usually  does  not  provide  sufficient  direction 
to  the  oontraotor  for  correcting  discrepancies.  The 
most  advisable  procedure  is  to  devote  considerable  atten¬ 
tion  to  evaluation  of  preliminary  drafts  of  the  program 
plan  which  progressively  incorporate  improvements  based 
on  eustosMr  recommendations  until  an  acceptable  over-all 
plan  is  achieved. 
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The  great  number  of  factors  to  be  considered  in 
evaluating  a  program  plan  suggests  that  an  objective 
approach  be  taken.  The  evaluation  can  never  be  entirely 
objective,  however,  since  several  qualities  such  as 
clarity  of  the  plan  and  understanding  of  requirements 
must  necessarily  be  reviewed  on  a  subjective  basis. 


Table  2-18  illustrates  one  approach  to  program  plan 
evaluation  --  a  tabulation  method  which  enables  recording 
of  certain  characteristics  relative  to  the  plan  and  notation 
of  Judgments  made  during  the  evaluation  process.  The  in¬ 
formation  required  for  completion  of  this  table  is  dis¬ 
cussed  on  the  basis  of  column  headings  as  follows: 


(1)  Subject  —  These  descriptive  titles  define 
the  program  plan  elements  which  are  to  be 
evaluated.  They  can  be  listed  prior  to 
actual  evaluation  of  the  plan,  since  in  most 
cases  they  will  represent  an  outline  of  the 
contract  statement  of  requirements. 

(2)  Weighting  Factor  —  This  numerical  value 
represents  a  decision  by  the  SPO  Reliability 
Coordinator  as  to  the  relative  importance 

of  the  subject  requirement.  Usually  an 
elaborate  weighting  factor  Bystem  is  not 
practical  since  virtually  all  requirements 
are  essential  to  the  success  of  the  program. 
However,  there  will  be  certain  requirements 
(such  as  demonstration  or  failure  reporting) 
which  warrant  identification  as  items  of 
particular  significance.  Any  weighting  factors 
used  should  be  assigned  prior  to  evaluation 
of  a  contractor's  program  plan. 


(3)  Statement  of  requirements  reference  paragraph  — 
The  number  of  the  paragraph  within  the  oontraot 
statement  of  requirements  (or  dooument  referenced 
within  the  contract)  which  covers  the  subjeot 
requirement  should  be  indicated. 

(4)  Contractor's  program  plan  reference  paragraph  — 
The  purpose  of  the  information  in  columns  (3) 
and  (4)  is  to  provide  cross-references  between 
the  oontraot  and  the  contractor's  program  plan 
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and  thus  facilitate  evaluation  of  the  responsive¬ 
ness  of  the  plan  to  the  contract  requirements. 

(5)  Adequacy  of  responsibility  assignment  —  A 
standard  requirement  for  a  reliability  program 
plan  is  the  assignment  of  specific  responsibilities 
within  the  contractor's  organization.  Hence,  the 
plan  should  contain  precise  statements  or  tabular 
presentations  which  fulfill  this  requirement. 
Compliance  or  non-compliance  with  the  requirement 
can  be  recorded  in  column  (5). 

(6)  Adequacy  of  procedures  —  Possibly  the  most 
important  characteristic  of  the  program  plan 
is  the  statement  by  the  contractor  as  to  the 
manner  of  fulfilling  contract  requirements. 

Hence,  the  SPO  Reliability  Coordinator  should 
compare  the  contractor* a  statements  with  thore 
in  the  contract  to  determine  the  adequacy  of 

the  contractor's  planned  procedures.  'Vny  special 
instructions  related  to  a  particular  element  snould 
also  be  used  as  a  standard  in  formulating  judg¬ 
ment.  In  general,  the  contractor's  reliability 
plan  should  not  be  confined  to  a  restatement  of 
the  contract  requirements  but  should  amplify 
them  to  the  extent  that  his  plan  represents 
a  positive  course  of  action. 

(7)  Adequacy  of  scheduling  —  In  the  treatment  of 
each  program  requirement,  there  will  normally 
be  at  least  one  ''milestone"  or  key  event.  Some 
requirements,  such  as  prediction  or  design  review, 
will  usually  be  characterized  by  several  key  events. 
These  events  should  be  scheduled  within  the  program 
plan  to  the  extent  possible,  and  the  SPO  Reliability 
Coordinator  should  Judge  the  adequacy  of  the 
scheduling  information  presented  by  the  contractor. 

(8)  Itemization  of  discrepancies  --  Columns  (5)#  (6), 
and  (7)  involve  either  positive  or  negative 
Judgments.  If  a  negative  Judgment  is  made,  it  is 

^  advisable  to  describe  the  specific  discrepancy  in 
column  (8).  This  action  permits  the  SPO  Reliability 
Coordinator  to  complla  a  series  of  comments  for 
the  contractor. 

(9)  Acceptability  --  This  column  should  be  used  as  an 
over-all  record  of  the  acceptability  of  elements 
within  the  contractor's  reliability  program  plan. 
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Table  2-18  has  been  filled  In  as  It  might  appear 
relative  to  evaluation  of  a  hypothetical  reliability 
program  plan.  The  subjects  listed  in  column  (l)  were 
arbitrarily  selected  for  the  purpose  of  Illustration. 


When  the  evaluation  of  a  particular  reliability 
program  plan  has  been  completed,  the  contractor  should 
be  advised  of  the  results  and  of  the  specific  actions  he 
must  take  to  obtain  acceptance  of  the  program  plan. 


2. 3. 2. 5  Problems  and  Other  Special  Considerations 
Relating  to  Program  Plan  Evaluation 


2. 3.2. 5.1  Use  of  Evaluation  Results 


The  results  of  the  evaluation  of  a  proposed  program 
plan  can  produce  benefits  which  are  not  necessarily  con¬ 
fined  to  one  program.  They  should  be  used  as  the  basis 
for  modifying  the  evaluation  procedure, developing  better 
program  plans.  Further,  because  a  program  plan  defines 
a  contractor's  reliability  activities  and  associated 
efforts  so  thoroughly,  some  relationship  should  exist 
between  adequacy  of  a  program  plan  and  ultimate  reliability 
achievements.  This  relationship  merits  eventual  Investiga¬ 
tion. 


2. 3.2. 5.2  Availability  of  Specialists 


Since  a  reliability  program  plan  may  Involve  technical 
items  requiring  unique  areas  of  knowledge,  specialists  should 
be  available  to  assist  the  SPO  Reliability  Coordinator  In 
performing  the  program  plan  evaluation.  The  appropriate 
use  of  specialists  can  make  the  evaluation  more  accurate. 

A  description  of  the  assistance  and  services  available  through 
the  SB/TD  contractors  is  presented  in  3.3*2. 


2.3. 2. 5.3  Program  Size  and  Complexity 

The  evaluator  of  a  program  plan  should  oonsider  the  size 
and  complexity  of  the  program  when  reviewing  the  plan.  The 
more  oomplex  the  program,  the  more  oarefully  the  relationships 


between  the  various  program  elements  and  schedules  must 
be  described  and  documented. 


2. 3.2. 5.4  Bias  in  Evaluating  Program  Plan 


There  is  a  possibility  of  unintentional  bias  during 
the  evaluation  of  a  program  plan.  This  problem  can  be 
minimized,  however,  by  the  use  of  more  than  one  evaluator. 


2. 3. 2. 5. 5  Re evaluation  of  Program  Plans 


Proposed  program  plans  will  sometimes  be  reevaluated 
after  additions  or  modifications  have  been  made.  Changed 
sections  or  new  sections  should  be  evaluated  as  though 
originally  proposed.  The  entire  program  plan  should  also 
be  reviewed  to  determine  that  previously  acceptable  sections 
are  still  satisfactory. 


2. 3.2.6.  Additional  References  on 

Evaluation  of  Contractor's  Program  Plan 


Reliability  and  Product  Assurance,  R.  L.  Landers, 
Prentice-Hall,  1963*  PP.  I2b-l5? . 


Reliability:  Management.  Methods, and  Mathematics 

D.  K.  Lloyd  and  M.,  Lipow,  Prentice-Hall,  1962 
PP.  34-35. 


3.  ADMINISTRATION  OP  A  RELIABILITY  PROGRAM 


Once  a  reliability  program  has  been  established 
(see  section  2),  the  administration  of  the  program, 
through  sound  management  actions,  becomes  a  major  task 
within  the  purview  of  the  SPO  Reliability  Coordinator. 
Although  the  activities  associated  with  the  terms 
"management"  and  "administration"  are  often  broad  in 
scope,  administrative  duties  must  not  be  random  but  must 
be  founded  on  an  ordered  set  of  directives,  policies  or 
procedures.  Section  3  is  intended  to  define  the  tasks 
encompassed  by  reliability  program  administration,  with 
specific  reference  to  the  responsibilities  of  the  SPO 
Reliability  Coordinators. 

Reliability  engineering  as  an  accepted  discipline 
developed  rather  recently,  although  the  mathematical  and 
statistical  methodology  required  In  reliability  research 
had  been  established  for  some  time.  During  the  1950's, 
two  types  of  reliability  research  began  to  be  emphasized: 

(l)  data  collection  or  fact-finding  effort  in  which  the 
primary  objective  was  the  accumulation  of  failure-rate 
information  through  testing  programs,  and  (2)  the  develop¬ 
ment  of  techniques  by  which  the  methods  of  mathematics 
could  be  applied  in  the  solution  of  reliability  problems. 
These  types  of  research  are  continuing  as  important  ele¬ 
ments  in  reliability  engineering,  but  they  have  lately 
been  Joined  by  another  type  of  research  which  had  pre¬ 
viously  received  little  attention  --  the  development  of 
methods  for  reliability  management. 

The  management  problem  presented  by  reliability  engi¬ 
neering  is  usually  defined  in  negative  terms,  with  emphasis 
on  the  item  that  fails,  the  mode  of  failure,  the  human 
errors  contributing  to  failure,  and  the  control  actions 
that  must  be  taken  to  prevent  failure  recurrence.  Although 
this  description  is  sound  and  can  be  used  effectively,  there 
is  another  more  affirmative  method  of  defining  the  problem: 
What  are  the  management  methods  that  promote  an  atmosphere 
in  which  maximum  product  quality  can  be  achieved?  By  empha¬ 
sizing  prevention  rather  than  correction  of  trouble,  this 
approach  compels  reliability  management  to  exercise  con¬ 
structive  responsibilities  rather  than  merely  remedial 
actions . 
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The  positive  approach  to  reliability  program  adminis¬ 
tration  has  been  emphasized  on  many  occasions  by  high  Air 
Force  authority.  For  example,  in  his  keynote  address  be¬ 
fore  the  Seventh  Military- Indus try  Missile  and  Space  Relia¬ 
bility  Symposium,  General  B.  A.  Schriever  stated: 

"We  can  no  longer  afford  to  take  the  easy  view, 
that  reliability  is  something  that  Just  happens. 

It  must  be  planned  for  and  worked  for  —  in  a 
careful,  organized,  and  systematic  manner.  In 
systems  acquisitions  today,  reliability  is  more 
than  Just  a  technical  problem  —  it  is  a  definite 
responsibility  of  management." 

The  emphaois  on  an  organized  and  systematic  approach  to  pro¬ 
gram  administration  is  further  manifest  in  existing  regu¬ 
lations  which  have  been  issued  at  all  levels  of  Air  Force 
command . 

From  the  Air  Force  standpoint,  the  responsibility  for 
reliability  management  is  primarily  vested  in  the  SPO 
Reliability  Coordinators.  Section  3  discusses  the  tasks 
of  a  Reliability  Coordinator  as  they  relate  to  administra¬ 
tion  and  management  of  an  established  reliability  program. 
Hence,  it  is  assumed  that  a  contract  has  been  awarded  and 
that  a  reliability  program  plan  exiBts  in  an  approved  form. 
The  function  of  the  Reliability  Coordinator,  then,  is  to 
take  those  actions  necessary  to  cause  implementation  of  the 
reliability  requirements  which  have  been  established  by  the 
contract  and  the  contractor’s  program  plan.  He  must 
assure  that  adequate  customer-contractor  communication 
channels  exist  for  mutual  understanding  of  program  require¬ 
ments  and  accomplishments  (3.1);  he  must  implement  efficient 
control  systems  or  devices  for  comparing  actual  and  planned 
progress  (3.2);  he  must  provide  effective  technical  direc¬ 
tion  to  Insure  program  progression  toward  planned  objectives 

(3.3) ;  he  must  endeavor  to  obtain  sufficient  funding  and 
other  support  required  for  proper  conduct  of  the  program 

(3.4) . 

The  task  of  monitoring  and  evaluation  of  a  reliability 
prograa  is  not  considered  in  Section  3*  tout  rather  is  sub¬ 
jected  to  speoial  consideration  in  section  4. 
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3.1  Customer^-Contractor  Communications 


Defined  as  a  task,  customer- contractor  communications 
encompass  all  activities  by  which  the  two  parties  inter¬ 
change  opinions,  recommendations,  decisions,  or  status 
information  to  increase  their  material  knowledge  or  under¬ 
standing  relative  to  a  particular  program. 

The  employment  of  effective  communication  techniques 
represents  a  major  challenge  in  the  administration  of  a 
reliability  program.  The  complexity  of  the  communication 
problem  can  be  attributed  to  the  following  factors  which 
are  characteristic  of  most  programs: 

(1)  The  technical  tasks  to  be  performed  by  a 
contractor  are  large  in  number  and  embrace 
many  areas  of  consideration. 

(2)  The  procedures  associated  with  the  practice 
of  reliability  engineering  are  not  routine 
and  therefore  require  continual  interpreta¬ 
tion  and  refinement. 

(3)  The  number  of  individuals  involved  in  the 
implementation  of  the  reliability  program  is 
large,  necessitating  extreme  measures  for 
coordination  of  effort. 

(4)  The  reliability  program  extends  over  a  con¬ 
siderable  period  of  time,  necessitating 
special  measures  to  maintain  continuity  of 
effort. 

(5)  Reliability  is  only  one  of  many  parameters  which 
are  vital  to  over-all  system  effectiveness  (see 
Appendix  P) .  Hence,  continual  trade-offs  are 
necessary  throughout  the  duration  of  the  program. 
Fob  these  trade-offs  to  be  made  properly,  up-to- 
date  information  on  program  progress  and  potential 
must  be  available  on  a  virtually  continuous  basis. 

Ihe  following  categories  of  communication  activity  are 
significant  to  the  administration  of  a  reliability  program: 

(1)  Periodic  status  reports  (from  the  contractor  to 
the  customer) . 

(2)  Periodic  technical  meetings  (of  both  contractor 
and  customer  representatives). 
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3.1.1  Periodic  Status  Reporting 


3. 1.1.1  Definition 


Periodic  status  reporting  Involves  the  submittal  by 
the  contractor  of  a  documented  statement  of  program  high¬ 
lights  (e.g.,  accomplishments,  problems  encountered, 
future  plans,  etc.)  to  the  customer  at  a  specified  fre¬ 
quency.  The  submittal  of  status  reports  usually  commences 
with  contract  award  and  continues  throughout  the  duration 
of  the  contract. 

This  reporting  activity  is  vital  to  the  administration 
of  a  reliability  program  since,  in  most  programs,  it  repre¬ 
sents  the  prime  method  by  which  the  contractor  communicates 
information  to  the  customer. 


3 . 1 . 1 . 2  Air  Force  Policy 


There  are  many  policy  statements  within  existing  Air 
Force  regulations  which  imply  that  periodic  status  report¬ 
ing  is  required  in  every  reliability  program.  As  a  typical 
example,  AFR  80-5  states  that  achievement  of  reliability  is 
a  growth  process  and  that  collection  of  information  is 
fundamental  to  this  process. 


3. 1.1. 3  Responsibilities  for  Status  Reporting 


The  assigned  responsibilities  of  the  System  Program 
Offices  (and  thus  the  SPO  Reliability  Coordinators)  and 
associated  groups  relating  to  administration  of  reliability 
status  reporting  for  contracted  programs  are  summarized  in 
Table  3-1. 


3.1.1.*l  Implementation  of  Status  Reporting 


The  taak  of  administering  the  contractor- to- customer 
status  reporting  activity  for  a  particular  program  encom¬ 
passes  three  distinct  areas  of  endeavor: 

(1)  Formulating  a  plan  for  report  submittals. 

(2)  Evaluating  submitted  reports  for  adequacy. 

(3)  Effectively  utilizing  reported  information. 
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TABLE  3-1 

SUMMARY  OF  RESPONSIBILITIES 
RELATING  TO 

ADMINISTRATION  OF  RELIABILITY  STATUS  REPORTING 


Air  Force 
Document 

Level 
to  Which 
Assigned 

Statement  of  Responsibility 

Par. 

No. 

AFSCR  80-1 

14  Dec.  1962 

AFSC  Division 
and  Centers 

(l)  Maintain  complete,  factual, 
and  timely  information  regarding 
the  status  of  reliability  pro¬ 
grams  as  well  as  associated 
problem  areas. 

6.f 

System 

Program 

Offices 

(SPO's) 

(l)  Maintain  complete,  factual, 
and  timely  information  regard¬ 
ing  the  status  of  reliability 
estimates  and  contractor's  relia¬ 
bility  program  as  well  as  associ¬ 
ated  and  program  problem  areas. 

7-h 

. 

(2)  Establish  the  minimum  amount 
of  detailed  reliability  program 
information  from  contractor  and 
supporting  Air  Force  agencies. 

(AFSCR  174-1  states  policies  to  be 
observed  in  establishing  reporting 
requirements  from  AFSC  contractors 
and  provides  procedures  to  be  fol¬ 
lowed  in  obtaining  approval  for 
requirements  for  contractor  re¬ 
ports  . )  Pertinent  reliability  in¬ 
formation  will  be  compiled  in 
periodic  reports  for  the  cognizant 
SPO.  The  information  contained  in 
these  reports  will  be  considered 
by  the  SPO  in  all  management  deci¬ 
sions  which  can  affect  the  achieved 
operational  reliability  of  the  sys¬ 
tem.  (Stated  as  a  procedure  in 

AFSCR  80-1.) 

9-a 

AFBSDR  80-5 
28  Dec.  1962 

SE/TD 

Contractors 

(l)  Assure  that  documents  necessary 
to  technical  management  of  the  sys¬ 
tem  reliability  program  are  gener¬ 
ated,  collected,  and  maintained. 

6.c.(4) 
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3. 1.1. 4.1  Formulation  of  Plan 

for  Report  Submittals 


For  maximum  effectiveness  to  be  realized  from  the 
status  reporting  activity,  both  the  customer  and  the 
contractor  must  have  a  clear  understanding  of  reporting 
specifics.  This  understanding  can  be  achieved  through 
contractual  stipulation  of  requirements,  amplification 
of  the  requirements  in  supplementary  formal  instructions, 
and  further  guidance  provided  by  informal  communication. 


3. 1.1. 4. 1.1  Contract  Reporting  Requirements 


Usually,  only  minimum  specifications  for  reporting 
are  stated  within  the  contract  instrument.  These  include: 

(1)  Title  of  the  report  (e.g..  Reliability  Status 
/Progress/  Report). 

(2)  Frequency  of  submittal  (e.g.,  monthly). 

(3)  General  scope  of  coverage  (e.g.,  report  of 
progress  relative  to  defined  reliability 
tasks  and  statement  of  significant  problem 
areas ) . 

(4)  Persons,  offices,  agencies,  etc.  to  whom  the 
report  is  to  be  submitted. 

In  some  cases,  the  contract  will  also  stipulate  com- 
pllance  with  an  existing  standard  or  instruction  for  report 
submittal  (e.g.,  AFBM  Exhibit  58-1).  This  standard  or 
instruction  will  normally  have  general  application  to  all 
reports  under  the  program  and  will  not  specifically  refer 
to  the  reliability  program  status  reports. 


3. 1.1. 4. 1.2  Supplementary  Reporting 
Instructions 


In  the  administration  of  a  particular  program,  it  will 
be  practical  and  even  necessary  for  the  SPO  Reliability  Co¬ 
ordinator  to  supplement  the  oontract  statement  and  its 
associated  general  instructions  for  reporting  with  a  specific 
set  of  instructions  for  periodic  reliability  status  reporting 


3.1-5 


"Instructions  for  Submittal  of  Monthly  Reliability 
tatus  Reports  —  Program  XXX").  Typloal  of  the  factors 
which  could  bs  included  in  such  instructions  are: 

(1)  Summary  of  oon tract  requirements  whieh  are 
pertinent  to  the  reliability  status  report 
(including  general  regulations). 

(2)  Date  of  month  (or  other  specific  time)  when 
the  report  is  to  be  submitted. 

(3)  Relationship  between  the  reliability  status 
report  and  the  over-all  program  status  report. 

It  is  important  to  establish  whether  relia¬ 
bility  status  will  be  Included  as  part  of  the 
over-all  program  status  report  or  will  be 
reported  separately. 

(4)  Basic  outline  for  the  presentation  of  the 
status  information.  Por  example ,  the  status 
report  oould  be  divided  into  discussions  for 
major  subsystems  or  for  other  defined  areas 

of  activity.  The  basic  outline  selected  should 
be  consistent  with  the  outline  of  program  re¬ 
quirements  as  established  by  the  contract  and/ 
or  the  contractor's  program  plan. 

(5)  A  statement  of  the  types  of  information  desired 
within  the  report,  with  a  description  of  each 
type.  Categories  of  desired  Information  could 
lnolude  quantitative  achievements;  significant 
events,  regardless  of  achievement  aspects; 
problem  areas;  listing  of  documents  generated 
for  the  reporting  period;  management  decisions 
made  by  the  contractor;  and  future  work  plans. 

(6)  Speoifio  forms  that  must  be  submitted  as  part 

of  the  status  report.  These  forms  may  be  specially 
designed  on  the  basis  of  individual  System  Program 
Of floes'  requirements 

(7)  Bxplloit  objective  statements  whloh  are  to  be 
lnoluded  in  the  report  as  a  matter  of  routine. 

Such  statements  could  lnolude  the  number  of 
eaoh  of  the  following  items  aooompllshed  dur¬ 
ing  the  reporting  period:  design  reviews, 
visits  to  subcontractor  facilities,  classroom 
hours  spent  on  reliability  training,  failure 
reports  prooessed. 


(8)  Relationship  between  the  periodic  reliability 
status  report  and  other  types  of  submittals 
associated  with  the  reliability  program.  The 
contract  and  the  program  plan  will  usually 
identify  several  other  distinct  submittals, 
such  as  prediction  reports,  failure  summaries, 
and  design  review  reports.  It  is  necessary  to 
indicate  whether  these  specialized  reports 
should  be  submitted  with  the  status  report  or 
separately. 

In  most  situations,  it  will  be  desirable  to  consider 
the  specifics  of  status  reporting  shortly  after  contract 
award.  This  timing  permits  the  contractor's  program  plan 
as  well  as  any  suggestions  or  opinions  he  may  have  relat¬ 
ing  to  report  preparation  to  be  taken  into  consideration 
when  preparing  the  detailed  instructions  for  reporting. 

The  SPO  Reliability  Coordinator  should  also  consider  his 
own  requirements  for  submitting  internal  Air  Force  reports 
(see  3.1.5)  and  make  certain  that,  where  appropriate, 
requirements  stipulated  by  higher  authority  are  passed 
on  to  the  contractor. 


Figure  3-1  illustrates  a  typical  manner  of  expressing 
reporting  requirements  for  a  program.  The  material  shown 
was  extracted  from  a  Sample  Work  Statement  issued  by  BSD 
for  guidance  to  procuring  activities. 


3. 1.1. 4. 1.3  Informal  Communication 
Relating  to  Reporting 


The  preceding  discussions  have  related  to  establishment 
of  reporting  specifics  through  contract  statements  and  sup¬ 
plementary  instructions  which  are  officially  issued.  In 
many  cases,  the  effectiveness  of  status  reporting  can  be 
further  increased  by  informal  communication  with  the  con¬ 
tractor  to  provide  additional  guidance  and/or  request  empha¬ 
sis  on  areas  of  immediate  interest. 


3. 1,1.' 4. 2  Evaluation  of  Reports 


Baoh  reliability  status  report  submitted  by  the  eon- 
traotor  should  be  evaluated  by  the  SPO  Reliability  Co¬ 
ordinator.  The  first  few  reports  submitted  in  a  report 


FIGURE  3-1 

EXAMPLE  OF  TYPICAL  METHOD 
OF  STATING  REPORTING  REQUIREMENTS* 


13.2  Sample  formats  for  these  reports  are  given  in 
Figures  2  through  7  on  the  following  pages. 

13.3  Other  special  or  one-time  reports,  including 
those  listed  below,  have  no  special  requiremente  regard¬ 
ing  format . 

C.l  Program  Review  Reports 

C.2  Interface  Definition  Report 

C.3  Design  Review  Packages  (on  request  only) 

C.4  Test  Review  Board  Report 

"Seotlon  13  from  Sample  Work  Statement 
iaeued  by  BSD. 


FIGURE  3-1  (Continued) 
EXAMPLE  OF  TYPICAL  METHOD 
OF  STATING  REPORTING  REQUIREMENTS* 


FIGURE  2 


A.l  Reliability  Estimates  (Furnished  by  IACC  from  AC  Inputs) 
System  _ Date  through  1963 


Requirement 
Count-Dn  Flight/Patrol 


Current  Status  Ct-Dn 
Fllght/Patrol _ 


A.  Over-all  System 

B.  Propulsion 

1.  1st  Stage 

2.  2nd  Stage 

C.  Guidance 

1.  Airborne 

2 .  Ground 

D.  Re-Entry  Vehicle 

E.  Command  and  Control 

1.  TLV 

2.  COC 

F.  Transporter  Launcher 

1.  ECU 

2.  APU 


-/- 


-/- 


FIGURE  3 

A. 2  Availability  Estimates 


Availability  MTTR (Unached)  MPT  (Sched) 


A.  Over-all  System 

B.  Propulsion 

C.  Guidance 

D.  RA 

E.  C  and  C 

1.  TLV 

2.  COC 

F.  T/L 


X 

X 


•Section  13  from  sample  Work  Statement 
issued  by  BSD 
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Cumulative  Number 
of  F/R _ 

Total  Number  of  P/ 
this  Month 


I 


FIGURE  3-1  (Continued) 

EXAMPLE  OP  TYPICAL  METHOD 
OP  STATING  REPORTING  REQUIREMENTS* 


A  s  s  emb ly/Che  c  k- Ou  t 
Other 
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EXAMPLE  OF  TYPICAL  METHOD 
OF  STATING  REPORTING  REQUIREMENTS* 


FIGURE  5 

B.l.  MATHEMATICAL  MODEL 


/ 

/  OVER  SYSTEM  REL 

/  BALANCE/ AVAILABILITY 

/  BAL. 


♦♦Reference  code:  This 
code  allows  one  to 
locate  on  back-up  sheets 
the  specific  five  (5) 
components  (next  lowest 
indenture)  which  are  fur¬ 
thest  from  their  appor¬ 
tioned  reliability. 


J t 


Reliability  Balance 

:-X(c  +  Dn/Flt) 

Function  Number 

:  3.2.1 

Availability  Estimate 

:  Y* 

Validated 

:  0  or  1 

Reference  Code 

:a*k,J,p,w 

Design  Reviewed 

:N/R  or  0  or  1 

♦Section  13  of  Sample  Work  Statement 
issued  by  BSD 
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FIGURE  3-1  (Continued) 

EXAMPLE  OF  TYPICAL  METHOD 
OF  STATING  REPORTING  REQUIREMENTS  * 


60 

u 

<b 


’g. 

-h. 


FIGURE  6 

B.2.  ZERO-TREND  ITEMS 

No.  of  Items  lacking  data  or  to  be  tested 

No.  of  Items  on  critical  parts  list 

No.  of  Items  which  do  not  have  failure  modes 
Identified 

No.  of  items  below  apportioned  reliability  goal 

No.  of  design  reviews  held 

No.  of  designs  released  (Specific  items  re¬ 
ceiving  program  office  approval  of  design 
reciew  will  be  described  and  identified  on 
back-up) 

No.  of  items  whose  environments  are  not  defined 

No.  of  'Buy'  items  which  do  not  have  specifica¬ 
tions  written 

No.  of  specifications  released  which  do  not 
contain  quantitative  reliability  requirements 

No.  of  test  objectives  remaining  to  be  accomplished 

No.  of  failures  requiring  failure  analysis 


f-  i  I  ii-  i — i — \ 


123  4  123^ - 

1964 

Calendar  Quarters 


•Section  13  of  Sample  Work  Statement 
issued  by  BSD 
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FIGURE  3-1  (Continued) 


EXAMPLE  OF  TYPICAL  METHOD 
OF  STATING  REPORTING  REQUIREMENTS* 


should  receive  particular  attention  to  assure  that  an 
understanding  exists  between  the  contractor  and  the  cus¬ 
tomer  as  to  the  needs  to  be  fulfilled  by  the  report. 

The  review  and  evaluation  of  status  reports  may 
involve  a  subjective  approach  relating  to  the  auality 
of  reporting  on  major  elements  of  the  program  (the  con¬ 
tractor  should  be  advised  in  advance  of  the  SPO's  con¬ 
cept  of  what  constitutes  a  major  element).  Or,  a  more 
objective  approach  utilizing  pre-established  check  list 
items  such  as  those  presented  in  Table  3-2  may  be  employed. 

When  the  SPO  Reliability  Coordinator  has  completed 
the  evaluation  of  a  status  report,  the  results  should  be 
communicated  to  the  contractor.  Such  communication  may 
be  a  matter  of  record  or  on  an  unofficial  basis;  however, 
recording  of  results  is  usually  more  effective  in  initi¬ 
ating  corrective  actions  by  the  contractor. 


3. 1.1. 4. 3  Utilization  of  Reported  Information 


If  a  sincere  effort  is  made  early  in  a  program  to 
assure  that  the  contractor  understands  and  implements 
the  requirements  and  desired  quality  of  reporting  effort, 
his  periodic  status  reports  can  serve  the  SPO  Reliability 
Coordinator  in  a  variety  of  ways.  For  maximum  value,  of 
course,  the  reports  must  have  a  consistent  organization 
and  display  continuity  with  respect  to  the  information 
reported. 

Specific  uses  which  the  status  reports  can  serve 
Include : 

(1)  Providing  the  major  source  of  information  for 
the  SPO  Reliability  Coordinator's  status  re¬ 
ports  to  his  higher  authority  (see  3.1.5). 

(2)  Providing  a  listing  of  action  items  for  dis¬ 
cussion  at  technical  direction  meetings 
(see  3.1.2) . 

(3)  Enabling  the  SPO  to  Initiate  proper  support 
activities  to  solve  existing  problems. 

(4)  Assisting  SPO  Judgment  as  to  the  adequacy  of 
the  contractor's  performance. 


3.1-14 


TABLE  3-2 

ITEMS  TO  BE  INCLUDED  IN  CHECK  LIST 
FOR  EVALUATION  OF  STATUS  REPORTS 


(1)  Is  the  report  submitted  on  time? 

(2)  Is  the  report  signed  by  the  appropriate  corporate 
official? 

(3)  Does  it  identify  the  reporting  period? 

(4)  Is  the  distribution  of  the  report  considered  adequate? 

(5)  Does  the  report  answer  questions  implied  by  the 
preceding  one? 

(6)  Does  it  identify  the  questions  to  be  answered  in  the 
next  report? 

(7)  Is  the  report  organization  consistent  with  specified 
format? 

(8)  Does  the  report  properly  Identify  included  references 
to  other  documents? 

(9)  Does  the  report  cover  all  items  scheduled  for 
completion  during  the  reporting  period? 

(10)  Is  the  information  in  sufficient  detail  to  enable 
the  required  level  of  understanding? 

(11)  Is  the  data  consistent  with  those  previously  reported? 

(12)  Is  the  report  in  sufficient  detail  to  enable  the  SPO 
Reliability  Coordinator  to  compile  his  own  internal 
reports? 

(13)  Are  all  major  aspects  of  the  program  covered? 

(14)  Does  the  report  Identify  significant  problem  areas? 

(15)  Does  it  identify  significant  events  or  findings  which 
are  truly  indicative  of  quantitative  reliability 
achievement? 

(16)  Has  the  contractor  complied  with  supplemental  report¬ 
ing  instructions  issued? 


(5)  Providing  information  for  updating  charts  or 
other  visual  displays  which  may  be  utilized 
in  the  program. 

(6)  Maintaining  a  historical  record  of  significant 
program  events . 


3. 1.1. 5  Problems  and  Special  Considerations 
Relating  to  Status  Reporting 


3. 1.1. 5.1  Relation  of  Program  Size 
to  Reporting  Frequency 


The  magnitude  of  the  contracted  program  will  determine, 
to  a  large  extent,  the  frequency  with  which  reports  should 
be  submitted.  Usually,  the  optimum  frequency  for  submittal 
is  on  a  monthly  basis,  but  in  programs  which  do  not  include 
comprehensive  effort,  status  reports  could  be  submitted 
less  frequently.  It  should  be  noted,  however,  that  MIL-R- 
27 542* requires  that  status  reports  are  to  be  submitted  at 
intervals  which  do  not  exceed  three  months. 


3. 1.1. 5 .2  Relation  of  System  Development  State 
to  Required  Status  Information 


The  state  of  development  of  the  system  will  be  influ¬ 
ential  in  determining  the  type  of  reliability  status  in¬ 
formation  to  be  reported.  For  example,  information  reported 
during  early  design  development  will  emphasize  specification 
and  drawing  preparation  activities,  prediction  accomplish¬ 
ments,  vendor  selection,  parts  improvement  programs,  train¬ 
ing  activities,  and  similar  developmental  efforts.  Later  in 
the  program,  test  results  will  be  more  readily  available, 
and  failure  reporting  activities  will  become  more  significant 
During  production  phases,  more  emphasis  will  be  placed  upon 
acceptance  techniques  and  field  results . 
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3. 1.1. 5. 3  Time  Consumption  In  Report 
Preparation 


It  Is  often  debated  that  time  consumed  In  preparing 
status  reports  shortens  the  available  time  and  therefore 
detracts  from  a  contractor's  ability  to  accomplish  tech¬ 
nical  elements  of  work.  Such  an  argument  usually  cannot 
be  Justified,  but  it  is  possible  that  the  contractor  could 
accomplish  the  reporting  task  more  efficiently.  Emphasis 
should  be  placed  on  establishing  a  format  which  facilitates 
reporting  needed  information.  Tabular  presentations,  graphs 
or  other  illustrations  will  simplify  reporting  of  data  and 
can  be  designed  for  cumulative  presentation  of  information 
from  each  reporting  period. 


3. 1.1. 5. 4  Coordination  Between  Reporting 
Procedures  and  Program  Needs 


It  is  usually  impractical  to  enforce  a  general  re¬ 
porting  procedure  without  specific  identification  of 
program  needs.  The  SPO  Reliability  Coordinator  may  hot, 
however,  be  in  a  position  to  fully  determine  these  needs 
early  in  a  program.  Hence  it  is  advisable  for  him  to 
invite  suggestions  from  the  contractor  in  formulating  de¬ 
tailed  specifications,  for  status  reporting. 


3. 1.1. 6  Additional  References 

on  Periodic  Status  Reporting 


AFBM  Exhibit  38-1,  Contractor  Report  Requirements. 

The  Computer  Reliability  Report,  I.  R.  Whiteman, 
Proceedings,  Ninth  National  Symposium  on  Relia¬ 
bility  and  Quality  Control,  January  1963#  pp.  80-83. 

Reliability  and  Product  Assurance,  R.  R.  Landers, 
Prentlce-Hairrilfei:  PP.  4 feb'-TO. 


Ttohnloal  Communication,  George  Harwell,  MacMillan, 
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Periodic  Technical  Meetings 


3. 1.2.1  Definition 


Periodic  technical  meetings  comprise  those  Joint 
meetings  of  appropriate  contractor  and  customer  personnel 
for  the  purpose  of  presenting  Information  or  discussing 
factors  significant  to  a  particular  reliability  program. 
These  meetings  provide  a. vehicle  for  open  treatment  of 
problem  areas  and  creation  of  a  better  understanding  of 
mutual  needs.  (Meetings  which  are  specifically  intended 
for  formal  review  of  program  progress  are  discussed  In  4.2.) 


Conduct  of  technical  meetings  in  an  informal  atmosphere 
is  most  conducive  to  beneficial  results.  However,  there 
are  definite  procedures  which  should  be  considered  In  im¬ 
plementing  this  area  of  activity,  including  specified 
schedules  and  pre-planned  agenda  for  meetings.  Technical 
meetings  between  contractor  and  customer  representatives 
should  commence  shortly  after  contract  award  and  continue, 
as  needed,  throughout  the  duration  of  the  program. 


3. 1.2. 2  Air  Force  Policy  and  Responsibilities 
for  Technical  Meetings 


Air  Force  reliability  directives  do  not  provide  explicit 
policy  statements  or  responsibility  assignments  relating  to 
the  conduct  of  technical  meetings.  However,  such  meetings 
are  standard  practice  in  the  management  function  for  a 
system  program,  and  SPO  Reliability  Coordinators  have  major 
functions  in  their  conduct,  as  indicated  in  the  following 
discussion  on  implementation 


3.1. 2. 3  Implementation  of  Technical 
Meetings 


In  conducting  a  series  of  customer-contractor  technical 
meetings  for  a  particular  program,  there  are  three  definable 
tasks  which  must  be  implemented t 


(1) 

Pre-planning  activities 

(2) 

Conduct  of  meetings 

(3) 

Reporting  of  results 

3. 1.2. 3. 

1  Pre-Planning  Activities 

The  conduct  of  technical  meetings  requires  the  presence 
of  key  personnel  from  both  the  contractor's  and  the  cus^ 
tomer's  organizations.  In  order  to  minimize  expenditure 
of  valuable  management  time,  special  attention  must  be 
given  to  the  efficient  planning  of  these  meetings.  Items 
to  be  considered  include: 

(1)  Frequency  of  meetings  —  The  intervals  at  which 
technical  meetings  should  be  held  can  be  pre¬ 
scribed  within  the  contract  statement  of  require¬ 
ments  or  established  Bhortly  after  contract  award. 
In  past  programs,  such  meetings  have  been  held 

as  frequently  as  on  a  monthly  basis,  but  the  need 
for  a  particular  program  depends  upon  the  extent 
of  reliability  activity  involved. 

(2)  Personnel  in  attendance  —  Because  of  the  joint 
contractor-customer  participation,  the  technical 
meetings  are  an  effective  means  for  generating 
significant  decisions  relative  to  the  reliability 
program.  Hence,  it  is  desirable  that  the  attend¬ 
ing  contractor  and  customer  personnel  be  of  appro¬ 
priate  management  levels  to  possess  authority  for 
implementing  such  decisions. 


(3)  Purpose  of  meetings  —  Technical  meetings  are 

usually  conducted  in  addition  to,  and  independent 
of,  periodic  status  reporting.  Therefore,  they 
should  preferably  not  be  used  to  report  routine, 
progress  but  rather  to  facilitate  the  identifica¬ 
tion  of  significant  problem  areas  and  formulate 
decisions  requiring  considerable  oustomer-oontractor 
coordination. 
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3. 1.2. 3. 2  Conduct  of  Meetings 


Universally  effective  meeting  procedures  are  applicable 
to  the  technical  meetings  associated  with  reliability  pro¬ 
grams!  a  chairman  (e.g.,  the  SPO  Reliability  Coordinator) 
should  be  appointed  to  direct  the  proceedings,  and  pre¬ 
planned  agenda  should  be  used. 


Undoubtedly,  during  the  course  of  a  technical  meeting, 
problems  will  be  introduced  which  warrant  additional  in¬ 
vestigation.  These  problems  could  be  either  technical  or 
administrative  in  scope  and  represent  areas  of  prime  respon¬ 
sibility  of  either  the  contractor  or  the  customer.  When 
such  problems  cannot  be  resolved  at  one  meeting,  they  should 
be  identified  as  "open  action"  items  to  be  discussed  further 
at  some  subsequent  meeting.  The  description  of  an  "open 
action"  item,  the  person  responsible  for  its  investigation, 
and  the  time  of  subsequent  reporting  should  be  recorded. 


3.1.2. 3. 3  Reporting  of  Results 


Subsequent  to  the  conduct  of  each  technical  meeting, 
the  minutes  should  be  published  and  distributed  to  all 
attendees.  Such  action  requires  that  a  secretary  be  appointed 
to  record  the  proceedings  while  they  are  in  progress.  The 
minutes  should  include  the  following  Information! 


(1)  A  summary  statement  of  the  topics  discussed. 

(2)  An  Identification  of  decisions  generated  during 
the  meeting. 

(3)  A  listing  of  "open  aotlon"  items  designated  for 
further  investigation. 


A  file  of  the  technical  meeting  minutes  for  a  particular 
program  should  be  maintained  by  the  SPO  Reliability  Coordina¬ 
tor,  This  file  will  serve  as  a  reference  for  historical  in¬ 
formation  and  for  future  planning  relative  to  program  ad¬ 
ministration. 


3. 1.2. 4  Problems  and  Special  Considerations 
Relating  to  Technical  Meetings 

3. 1.2.4. 1  Maintenance  of  Pre-Planned 
Schedule 


It  Is  not  uncommon  for  deviation  from  a  pre-planned 
meeting  schedule  to  occur  as  the  program  progresses,  due 
In  most  cases  to  the  difficulty  of  scheduling  meeting 
times  and  places  which  are  mutually  acceptable  to  all 
concerned.  There  Is  no  ready  solution  to  maintaining  a 
consistent  schedule  of  technical  meetings,  but  the  situa¬ 
tion  can  be  alleviated  by: 

(l)  Emphasizing  the  Importance  of  technical  meetings 
in  the  contract  statement  of  requirements. 


(2)  Pre-planning  each  meeting  by  the  advance  publi¬ 
cation  of  the  agenda. 

(3)  Requiring  attendance  of  key  personnel. 

(4)  Using  the  meetings  aB  a  vehicle  for  generating 
program  decisions. 

(5)  Properly  publishing  and  distributing  the  minutes 
of  meetings. 


3. 1.2.5  Additional  References 
on  Technical  Meetings 


The  Role  of  the  Buyer  in  Reliability.  R.  T.  Dewey, 
Proceedings,  Ninth  National  Symposium  on  Reliability 
and  Quality  Control,  January  1963. 
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3.1.3  Contractor  Data  Submittals 


3. 1.3.1  Definition 


The  various  data  utilized  by  a  contractor  to  predlot 
reliability,  establish  reliability  requirements,  estimate 
environmental  and  operational  effects,  estimate  aohleved 
reliability,  and  demonstrate  contract  compliance  also 
provide  means  for  the  oustomer  to  verify  and  evaluate 
decisions  and  performance  of  the  contractor.  Specific 
forms  of  data  presentation  are  normally  prescribed  within 
the  oontract  statement  of  requirements,  with  the  stipula¬ 
tion  that  these  data  be  submitted  to  proper  Air  Force 
authority.  In  most  Instances  Involving  elements  of  a 
reliability  program,  the  proper  authority  Is  the  SPO 
Reliability  Coordinator. 


3. 1.3. 2  Air  Force  Policy 


Air  Force  general  policy  and  requirements  relating 
to  oontraotor  data  submittals  are  presented  In  Table  3-3. 


3. 1.3. 3  Responsibilities  for  Contractor 
Data  Submittals 


The  assigned  responsibilities  for  System  Program 
Offices  and  other  associated  levels  of  authority  relat¬ 
ing  to  oontraotor  data  submittals  are  summarized  In 
Table  3~^*  The  administrative  responsibilities  of  the 
SPO  Reliability  Coordinator  are  primarily  related  to 
the  establishment  and  enforcement  of  orderly  proeedures 
for  aooompllshment  and  utilization  of  these  submittals. 


3.1.3.^  Implementation  of  Contractor 
Data  Submittals 


Tasks  to  be  implemented  by  the  SPO  Reliability  Co¬ 
ordinator  relative  to  oontraotor  data  submittals  are: 

(1)  Definition  of  data  submittal  requirements. 

(2)  Establishment  of  prooedure  for  review  and 
evaluation  of  data  submittals. 

(3)  Continuing  administration  and  oontrol  of 
oontraotor  data  submittal  aotlvltles. 


TABLE  3-3 


SUMMARY  OF  AIR  FORCE  POLICY 
RELATIVE  TO  CONTRACTOR  DATA  SUBMITTALS 


Air  Force 
Document 

Statement  of  Policy 

Par. 

No. 

AFR  80-5 

Achieving  reliability  is  a  growth  process. 

3-c 

4  June  1962 

The  collection,  analysis,  and  feedback  of 
information  to  both  the  Air  Force  and 
industry  are  fundamental  to  the  control 
of  this  process. 

MIL-R-27542A  1 

Data  as  referenced  or  described  in  the 

21  May  1963 

specification  under  the  following  para¬ 
graph  headings  shall  be  submitted  by 
bidders  or  contractors  at  such  times  as 
required: 

(l)  Reliability  Program  Plan 

3-3 

(2)  Program  Review 

3-4 

(3)  Emergency  Reporting  of  Defective 

Parts 

3. 5. 3-2. C 

(4)  Critical  Items 

3-5.4 

(5)  Design  Review 

3.5.10 

(6)  Reliability  Considerations  for 
Engineering  Changes 

3.5.11.5 

(7)  Failure  Data  Collection,  Analysis, 
and  Corrective  Action 

3-5.12 

(8)  Reliability  Demonstration 

3.5.13 

(9)  Periodic  and  Final  Reports 

3-5.14 
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TABLE  3-4 

SUMMARY  OP  RESPONSIBILITIES 
RELATING  TO  CONTRACTOR  DATA  SUBMITTALS 


Air  Force 
Dooument 

Level 
to  Whloh 

Assigned 

Statement  of  Responsibility 

Par. 

No. 

APR  80-5 

4  June  1962 

APSC 

(1)  Establish  procedures  for  failure 
data  feedback  and  insure  prompt  con¬ 
sideration  of  failure  data  for  incor¬ 
poration  into  systems  design  during 
researoh  and  development  and  for  use 
in  loglstlo  support  applications. 

6.1 

AFSCR  80-1 

14  Dec.  1962 

APSC 

Divisions 
end  Centers 

(1)  Establish  and  man  a  central  data 
offloe  for  the  collection,  evaluation, 
dissemination  of  applicable  reliability 
statistics  and  test  results  to  APSC  divi¬ 
sions  and  center  elements  and  furnish 
consultant  servloe  in  the  application 
of  this  information  to  new  programs. 

6.e 

System 

Program 

(1)  Incorporate  requirements  for  quanti¬ 
tative  data  in  contractual  documents. 

T-* 

Offices 

(SPO's) 

(2)  Maintain  complete,  faotual,  and 
timely  information  regarding  the  status 
of  reliability  estimates  and  contractor 
reliability  program  as  well  as  associated 
and  program  problem  areas. 

7-h 

APBSDR  80-5 

28  Dec.  1962 

SPO 

Reliability 

Coordinators 

(l)  Assume  oomplete  and  over-all  manage¬ 
ment  responsibilities  for  all  aspeots  of 
the  system  reliability  program  including: 

6.b.(3) 

(a)  Maintaining  a  current  status  of 
reliability  program  activities, 
achievements,  and  predictions. 

6.b.(3Xa) 

(b)  Reviewing  oontrao tor's  and  field 
failure  summaries,  isolating  defici¬ 
encies,  and  assuring  that  oorrfotive 
aotlon  is  taken  by  the  contractor  or 
other  responsible  agenoies. 

6.b.(7) 

SB/TO 

Contractors 

(l)  Evaluate  and  assure  that  data  and 
assumptions  used  by  the  oontraotor  are 
valid  prior  to  preparation  of  plans  for 
demonstration  of  aohleved  reliability. 

6. c. (3) 

(2)  Accumulate  and  evaluate  reliability 
data  from  whatever  souroes  are  available. 

6.c.(5) 

(3)  Prooess  failure  and  problem  summaries 
as  referenced  in  oontraotual  reliability 
documents . 

6.c.(6) 

t 


3. 1.3. 4.1  Requirements  for  Data  Submittals 


Requirements  for  contractor  data  submittals  should 
be  associated  with  specific  reporting  activities  In  a 
reliability  program  and  should  be  as  definitive  as  possible 
to  facilitate  control  of  the  quality,  content,  and  schedule 
of  submittals.  The  requirements  should  be  Incorporated  In 
the  contractor's  reliability  program  plan  or  other  related 
documents . 

Table  3-5  is  a  typical  check  list  which  dould  be  used 
by  the  SPO  Reliability  Coordinator  both  in  defining  data 
submittal  requirements  and  in  monitoring  compliance  with 
these  requirements.  The  report  and  documentation  titles 
listed  in  the  table  are  representative  of  those  which  might 
be  extracted  from  a  specific  contractual  statement  of 
requirements . 


3. 1.3. 4. 2  Review  of  Data  Submittals 


(~)  Each  data  submittal  should  be  subjected  to  a  review 

which  considers  the  following  questions: 

(1)  Are  the  data  pertinent  to  the  specific  purpose 
of  the  report  in  which  they  are  submitted  and 
to  the  conclusions  or  recommendations  in  the 
report? 

(2)  Are  the  sources  of  the  data  identified? 

(3)  Are  any  interpolations  or  extrapolations  made 
during  the  use  of’ the  data  identified  as  such? 

(4)  Are  the  applicable  operating  and  environmental 
conditions  adequately  described? 

(5)  Are  valid  assumptions  made  in  the  use  of  the 
data? 

(6)  Are  the  data  consistent  with  those  contained 
in  previous  reports? 

(7)  Are  the  data  sufficient  for  Air  Force  needs? 
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TABLE  3-5 

SAMPLE  CHECK  LIST 
FOR  CONTRACTOR  DATA  SUBMITTALS 


Required  Reliability  Report 
_ or  Documentation 

Reliability  Program  Plan 


Information 


Reliability  Status  Report 


Failure  Summary  Report 

Environmental  Requirements  Report 

Part  Selection  Report 

Product  Environmental  Test  Plan 

Product  Evaluation  Teat  Status 
Report 

Reliability  Requirements 
for  Subcontractors 

Reliability  Evaluation  Test  Plan 

Flight  Proofing  and 

Qualification  Status  Report 

Flight  Analysis  Report 

Detailed  Design  Study 
Reliability  Analysis 

Preprototype  Reliability  Analysis 

Prototype  Reliability  Analysis 

Preproduction  Reliability  Analysis 

Service  Readiness 
Reliability  Analysis 
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Product  Improvement 
Reliability  Analysis 
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(8)  For  test  data: 

(a)  Are  exclusions  rioted? 

(b)  Are  the  operating  conditions,  time,  and 
environments  (real  or  simulated)  identified? 

(9)  Do  graphs,  charts,  and  trade-off  curves  contain 
data  references  sufficient  to  identify  the  source? 

3. 1.3. 4.3  Administration  and  Control 
of  Data  Submittals 


The  continuing  administration  and  control  of  a  con¬ 
tractor’s  data  submittals  may  encompass  the  following 
activities : 

(1)  Maintenance  of  a  schedule  log  on  all  required 
reports  and  documentation,  and  notification 

of  the  contractor  whenever  reports  are  overdue. 

(2)  Maintenance  of  an  evaluation  log  on  all  reportB 
and  documentation,  including  remarks  on  data 
deficiencies,  discrepancies,  or  misapplications. 

(3)  Checking  of  each  report  received  from  the  con¬ 
tractor  to  determine  that  the  required  data  are 
present  and  satisfactory  (see  3.1. 3.4.2). 

(4)  Notification  of  the  contractor  as  to  any  dis¬ 
crepancies  or  deficiencies  in  the  data  or  their 
use  and  required  remedial  action. 

(5)  Periodically  requesting  Air  Force  Plant  Repre¬ 
sentatives  to  verify  data  submitted  by  the 
contractor  (e.g.,  in  failure  summaries),  note 
exclusions  and  questionable  items,  and  report 
the  results  of  their  investigations.  (Air  Force 
management  and  the  contractor  should  be  notified 
when  serious  deficiencies  exist.) 

(6)  Informing  the  contractor  within  a  reasonable  time 
after  SPO  review  or  contractor  oorreotive  action 
when  data  and  reports  are  considered  acceptable. 


s 

(7)  Taking  such  other  actions  as  may  be  required, 
for  example: 

(a)  Generating  Air  Force  reliability  studies. 

(b)  Initiating  system  or  subsystem  evaluation 
by  Air  Force  management. 

(o)  Initiating  action  required  by  Sg/TD 
contractors . 


0 
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3.1.4 


Customer-Contractor  Liaison 


3. 1.4.1  Definition 


Liaison  can  be  defined  as  the  informal  communicative 
efforts  between  the  customer  and  a  contractor  which  are 
intended  to  promote  better  understanding  of  requirements 
and  coordinate  activities  of  a  reliability  program.  Such 
efforts  include  the  informal  contacts  which  occur  during 
customer  visits  to  contractor  facilities.  Liaison  activities 
commence  upon  contract  award  and  continue  throughout  the 
program. 


3. 1.4.2  Air  Force  Policy 


The  policy  of  the  Air  Force  with  respect  to  liaison 
activities  is  summarized  in  Table  3-6. 


TABLE  3-6 

SUMMARY  OF  AIR  FORCE  POLICY  RELATING  TO  LIAISON  ACTIVITIES 


Air  Force 
Document 

Statement  of  Policy 

Par. 

No. 

AFSCR  80-1 

14  Dec.  1962 

AFBSDR  80-5 

28  Dec.  1962 

Procedures  for  periodic  management  review 
and  program  control  will  be  developed  and 
applied  to  the  management  activity  at 
individual  systems  management  level. 

4. h/ 

5.  a 

AFSCR  80-1 

14  Dec.  1962 

The  problems  resulting  from  time  compres¬ 
sion  of  programs  and  concurrent  activities 
can  be  resolved  only  by  exploiting  the 
systems  organization  to  achieve  an  inte¬ 
grated  approach. 

3-c 

3-1-29 


3. 1.4.3  Responsibilities  for  Customer- 
Contractor  Liaison 


The  responsibilities  of  SPO  Reliability  Coordinators 
and  other  levels  of  authority  relating  to  liaison  activities 
are  presented  in  Table  3-7. 


3. 1.4,4  Implementation  of  Liaison  Activities 


Liaison  activities  which  are  common  practice  in  the 
administration  of  reliability  programs  fall  into  two  dis¬ 
tinct  areas : 

(1)  Visits  to  contractor  facilities. 

(2)  Coordination  with  Air  Force  on-site 
surveillance  agencies  (e.g. ,  APPRO). 


3. 1.4.4. 1  Visits  to  Contractor  Facilities 


The  practice  of  visiting  a  contractor's  facility  offers 
obvious  advantages  to  the  SPO  Reliability  Coordinator*  He 
not  only  acquires  a  familiarity  with  the  contractor's  per¬ 
sonnel,  organization,  and  plant,  but  also  has  an  effective 
opportunity  to  promote  the  degree  of  personnel  interest 
which  the  contractor  manifests  in  the  conduot  of  his 
reliability  program. 


While  visits  to  contractor  facilities,  as  discussed 
here,  are  not  Intended  primarily  for  program  evaluation, 
they  nevertheless  provide  the  SPO  Reliability  Coordinator 
with  information  whloh  is  of  value  in  assessing  program 
progress  (see  4.1  for  discussion  of  use  of  visitation 
to  evaluate  programs ) .  Also,  through  a  first-hand  under¬ 
standing  of  problems  which  may  prevail  in  contractor  actl'- 
vltlea,  the  Reliability  Coordinator  can  more  effectively 
formulate  appropriate  solutions  or  provide  direction. 


TABLE  3-7 


SUMMARY  OF  RESPONSIBILITIES  RELATING  TO  LIAISON  ACTIVITIES 


Air  Force 
Document 

Level  To 
Which 
Assigned 

Statement  of  Responsibility 

Par. 

No. 

AFR  80-5 

4  June  1962 

AFSC 

Insure  uniform  implementation  of 
reliability  policy  and  procedures 
at  appropriate  levels  and  coordin¬ 
ate  with  related  activities 

6. a 

AFSCR  80-1 

14  Dec.  1962 

AFSC/DCS 

Establish  and  maintain  channels 
for  the  exchange  or  reliability 
information  with: 

(a)  Operating  commands  to  obtain 
information  for  the  establishment 
of  realistic  quantitative  require¬ 
ments. 

(b)  AFLC  to  obtain  reliability 
information  compiled  by  AFLC  and 
to  provide  information  from  which 
AFLC  may  program  adequate  support 
for  systems  becoming  operational. 

Data  exchange  channels  will  be 
established  with  any  military, 
industrial,  or  academic  agency 
which  requires  or  may  provide 
reliability  data. 

5. a. (2) 

AFSC 

Divisions 

and 

Centers 

Provide  technical  assistance  to 
the  CMR's  in  execution  of  their 
reliability  responsibilities. 

o.g 

System 

Program 

Offices 

(SPO's) 

Establish  a  liaison  with  AFLC  for 
developing  a  failure  reporting 
system  which  will  be  administered 
by  AFLC  during  system  testing  and 
the  timely  feedback  of  the 
generated  information  to  the 
development  engineering  activity 
and  the  central  data  office.  In 
addition,  the  SPO  will  assure 
the  delivery  to  AFLC  of  additional 
information  required  to  program 
AFLC  support  of  the  system  upon 
its  becoming  operational. 

T.g 

AFBSDR  80-5 

28  Dec.  1962 

SPO 

Reliability 

Coordinators 

Assure  proper  utilization  of  avail¬ 
able  manpower  resources  by  con- 
eumating  SPO/AFPRO-CMR  agreements 
and  validating  requirements  for 
SE/TD  support  as  necessary. 

6  b. (3) 

3-1-31 


» 

Coordination  with  Qn-Slte 
Surveillance  Agencies 


The  SPO  Reliability  Coordinator  should  Integrate  his 
administrative  activities  with  those  of  Air  Force  agencies 
having  related  responsibilities.  For  example.  Air  Force 
Plant  Representatives  who  report  to  the  Contract  Management 
Region  Office  are  stationed  at  the  facilities  of  most  large 
prime  contractors.  The  function  of  these  Plant  Represent¬ 
atives  Is  to  maintain  surveillance  during  development  and 
production  of  equipments  being  procured  by  the  Air  Force. 

In  most  cases,  an  AFPRO  will  Include  a  specific  reliability 
and/or  quality  control  group. 


Early  In  a  given  program,  the  SPO  Reliability  Coordin¬ 
ator  should  take  the  following  essential  actions: 

(1)  Initiate  contact  with  the  appropriate  AFPRO 
personnel  to  determine  their  local  practices 
and  procedures  and  to  coordinate  respective 
activities. 

(2)  Advise  the  AFPRO  of  the  specific  nature  and 
requirements  of  the  reliability  program  as 
contracted. 


The  results  of  this  early  coordination  effort  should' be 
stipulated  In  written  form  and  issued  as  a  joint  memorandum 
of  agreement. 


3.1.4. 5  Problems  and  Special  Considerations 
Relating  to  Liaison  Activities 


3.1*4. 5.1  Non-Interference  with  Contractor 
Operations 


It  Is  often  argued  that  frequent  visits  by  a  customer 
representative  reduce  the  contractor's  available  time  for 
performing  necessary  managerial  and  technical  functions. 

Such  an  opinion  by  a  manager  within  a  contractor's  organisa¬ 
tion  can  frequently  be  attributed  to  the  fact  that  the  manager 


assumes  an  obligation  to  prepare  for  customer  visits  and  . 
to  accompany  customer  representatives  during  the  duration 
of  such  visits.  There  will,  of  course,  be  certain  occasions 
when  a  SPO  Reliability  Coordinator's  visit  will  be  related 
to  an  official  purpose  (e.g.,  conduct  of  a  formal  program 
review  as  discussed  in  4  .2  ).  However,  on  other  occasions, 

the  Reliability  Coordinator  may  simply  desire  to  tour  the 
facility  unofficially  or  to  discuss  general  items  of  in¬ 
terest  in  the  program.  In  such  cases,  little  or  no  prepara¬ 
tion  by  the  contractor  is  necessary,  and  expenditure  of 
management  time  is  not  essential. 


The  SPO  Reliability  Coordinator  may  or  may  not  desire 
to  advise  the  contractor  in  advance  of  his  intention  to 
visit.  If  advance  notice  is  given,  the  purpose  of  the 
visit  should  be  clarified  and  the  specific  degree  of  pre¬ 
paration  (if  any)  that  is  anticipated  should  be  stated. 
This  procedure  will  minimize  non-essential  interference 
with  a  contractor's  operations. 


3. 1.4. 5. 2  Contact  with  Proper  Contractor 
Authorities 


During  visits  to  a  contractor's  facility,  a  SPO  Relia¬ 
bility  Coordinator  may  come  in  contact  with  a  large  number 
of  personnel.  It  is  essential  that  he  avoid  issuing  verbal 
directives  to  personnel  without  proper  reference  to  inter¬ 
mediate  contractor  authority.  In  every  program,  the  SPO 
Reliability  Coordinator  will  have  a  primary  point  of  con¬ 
tact  within  the  contractor's  organization  (usually  the 
reliability  manager  for  the  program).  He  must  establish 
and  maintain  official  communication  channels  with  this 
contact  and  issue  program  directives  accordingly.  Any 
directive  having  an  influence  upon  the  contract  must  he  formally  issued 
through  the  cognizant  contracting  officer. 

Verbal  directives  are  usually  not  proper  or  effective 
in  the  administration  of  a  program.  When  such  directions 
are  given  for  reasons  of  expediency,  they  should  be  sub¬ 
sequently  documented  as  soon  as  possible. 


3. 1.4. 5. 3  Relation  Between  SPO 
and  APPRO 


» 


When  difficulty  or  confusion  exists  between  the 
SPO  Reliability  Coordinator  and  the  APPRO  as  to  relation¬ 
ship  of  duties,  the  problem  can  usually  be  attributed  to 
lack  of  proper  coordination  effort  early  in  the  program. 
Hence,  the  SPO  Reliability  Coordinator  should  take  steps 
to  prevent  such  occurrence  by  proper  contact  with  the 
AFPRO  at  the  earliest  possible  convenience. 


3. 1.4.6  Additional  References 


on  Liaison  Activities 


Reliability  and  Product  Assurance,  R.  R.  Landers, 
Prentice-Hall,  1963,  PP.  449-452. 

The  Role  of  the  Buyer  in  Reliability,  R.  T.  Dewey, 
Proceedings,  Ninth  National  Symposium  on  Reliability 
and  Quality  Control,  January  1963. 


0 
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3.1.5  Customer  Internal  Reliability  Reports 


3. 1.5.1  Definition 


The  periodic  status  reports  discussed  in  3.1.1  pertain 
to  those  submitted  by  a  contractor.  In  addition  to  such 
reports,  the  System  Program  Office  is  required,  in  turn,  to 
periodically  report  the  status  of  reliability  effort  in  a 
contracted  program  to  higher  Air  Force  authority.  Require¬ 
ments  relating  to  the  preparation  and  submittal  of  these 
internal  reliability  reports  are  discussed  in  3.1.5. 


3. 1.5. 2  Air  Force  Policy 


Air  Force  policy  indicating  a  need  for  internal  re¬ 
porting  activity  is  typified  by  the  following  statement 
from  AFR  80-5:  "Achieving  reliability  is  a  growth  process. 
The  collection,  analysis,  and  feedback  of  information  to 
both  the  Air  Force  and  industry  are  fundamental  to  the 
control  of  this  process." 


3. 1.5. 3  Responsibilities  for  Internal 
Reliability  Reports 


Assigned  responsibilities  of  the  System  Program  Offices 
(and  thus  the  SPO  Reliability  Coordinators)  and  associated 
levels  of  authority  relating  to  internal  reporting  on  relia¬ 
bility  status  are  summarized  In  Table  3-8. 


3.1.5.^  Implementation  of  Internal 
Reliability  Reports 


The  general  responsibility  of  the  SPO  Reliability 
Coordinator  with  respect  to  internal  reliability  reporting 
is  to  screen  the  information  that  is  submitted  by  the  con¬ 
tractor  and  seleot  that  which  is  of  sufficient  significance 
to  pass  on  to  higher  Air  Force  management.  AFSCR  80-1  pre¬ 
scribes  in  detail  the  information  which  is  to  be  incorporated 
in  Internal  reliability  reports  and  the  procedures  for  pre¬ 
paring  such  reports.  Two  distlnot  types  of  reports  are  pre¬ 
scribed:  (1)  a  monthly  reliability  report  and  (2)  a  quarterly 
reliability  status  report. 


!  TABLE  3-8 

I  SUMMARY  OF  RESPONSIBILITIES 


RELATINO  TO  INTERNAL  REPORTING  WITHIN  AIR  FORCE 


Air  Porce 
Document 

Level 
to  Which 
Assigned 

Statement  of  Responsibility 

Par. 

No. 

AFSCR 

80-1 

1U  Dec.  1962 

APSC 

DCS/ 

Systems 

(l)  Prescribe  policies  and  procedures  as 
general  guides  to  be  followed  by  AFSC  systems 
divisions  and  centers  in  monitoring  and  re¬ 
porting  on  the  reliability  programs  for  the 
various  systems,  subsystems,  and  equipment 
under  development. 

5-a.(4) 

System 

Program 

Offices 

(SPO's) 

(1)  As  requested,  report  current  status  of 
reliability  programs  to  higher  management 
levels  of  the  Air  Force. 

(2)  Forward  all  major  reliability  problems 
which  cannot  be  resolved  by  the  SPO,  with 
the  recommended  actions,  including  implica¬ 
tions  for  operational  utilization  and  effec¬ 
tiveness,  to  the  appropriate  levels  of  Air 
Force  management. 

9-b 

9.c 

(3)  Define  major  reliability  program  check¬ 
points  or  milestones  on  both  activities  and 
results  (as  available)  and  integrate  them 
into  over-all  weapon  system  program  control 
procedures,  including  computer  or  other 
mechanized  procedures. 

9-d 

(4)  Provide  the  following  general  informa¬ 
tion  in  reliability  status  reports  to  higher 
Air  Force  authorities: 

9-e 

(1)  Requirements.  Status  of  actions  to 
incorporate  quantitative  requirements, 
requirements  for  demonstration,  a  com¬ 
prehensive  reliability  program  or  effort, 
and  reporting  on  that  effort  by  the  con¬ 
tractor  in  program  and  contractual  docu¬ 
ments  . 

9-e.(l) 

(2)  Reliability  Program.  Status  of  each 
major  reliability  activity  comprising 
the  system  contractor  efforts,  as  appro¬ 
priate,  including  adequacy  and  timeliness 
of  eaoh  activity. 

9.e.(2) 

(3)  Design  Considerations.  Status  of 
design  for  reliability,  including  tech¬ 
niques  employed  to  aohleve  system,  sub¬ 
system  and  component  reliability; 

- 

9-e.(3) 

TABLE  3-8  (Continued) 
SUMMARY  OP  RESPONSIBILITIES 


RELATING  TO  INTERNAL  REPORTING  WITHIN  AIR  FORCE 


Air  Force 
Document 

Level 
to  Which 
Assigned 

Statement  of  Responsibility 

Par. 

No. 

AFSCR  80-1 

14  Dec.  1962 

System 

Program 

Offices 

(SPO's) 

results  of  environmental  and  statistical 
testing,  and  the  degree  of  risk  in 
achieving  reliability  (including  identi¬ 
fication  of  all  critical  items).  When¬ 
ever  possible,  estimation  of  risk  will 
be  made  by  numerical  comparison  of  re¬ 
quirements  (failure  rates  or  other  appro¬ 
priate  measurement)  with  past  experience 
on  like  itemB.  In  every  case,  critical 
items  must  be  identified,  whether  by 
engineering  tests  and  Judgment  or  quanti¬ 
tative  analysis  of  data.  Unknowns,  such 
as  items  receiving  no  teBt  to  date,  will 
be  indicated. 

9-e. (3) 

(4)  Corrective  Actions.  The  statuB  of 
actions  taken  to  correct  or  alleviate 
critical  technical  and  program  problems 
by  contractor  and/or  Air  Force  agencies. 

9-e. (4) 

(5)  Systems  Analysis.  The  results  of 
systems  analysis  including  trade-offs 
between  reliability,  time,  cost,  sched¬ 
ules,  performance,  and  system  effective¬ 
ness.  Such  analysis  begins  In  the 
conceptual  phase  of  a  program,  is  pro¬ 
gressively  refined  as  system  character¬ 
istics  are  defined  in  more  detail,  and 
additional  data  becomes  available  as  a 
result  of  testing.  Explicit  definition 
of  all  factors  and  identification  of  all 
assumptions  should  be  one  of  the  early 
products  of  this  analysis. 

9-e. (5) 

(6)  Results.  The  prediction,  estimation, 
or  measurement  of  results  for  the  system, 
major  subsystems,  and  where  practicable, 
major  components.  Graphical  presentation 
of  results  is  desirable.  Confidence 
figures  or  other  numbers  indicating 
significance  of  the  results  must  be 
shown . 

9-e. (6) 
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TABUS  3-«  (Continued) 
SUMMARY  OP  RESPONSIBILITIES 


RELATING  TO  INTERNAL  REPORTING  WITHIN  AIR  FORCE 


Air  Force 
Document 

Level 
to  Which 
Assigned 

Statement  of  Responsibility 

AFSCR  80-1 
1U  Dec. 1962 

System 

Program 

Offices 

(SPO's) 

(7)  Significance  of  Results  from  an 
Operational  Standpoint.  Implications 
of  reliability  and  system  effectiveness 
for  operational  utilization  including, 
as  appropriate,  the  relationships  to 
such  factors  as  availability  or  incom¬ 
mission  rates,  target  allocation,  number 
of  systems  required,  maintenance  work¬ 
load,  and  requirements  for  spares  and 
ground  support  equipment.  par.  9.e.(7) 

3. 1.5. 4.1  Monthly  Reliability  Report 


Detailed  instructions  for  the  preparation  and  sub¬ 
mittal  of  the  Monthly  Reliability  Report  are  contained 
in  PMI  1-5. 


3. 1.5. 4.2  Quarterly  Reliability  Status  Report 


For  a  given  contracted  program,  the  System  Program 
Office  is  to  prepare  a  Reliability  Status  Report  at  the 
end  of  each  calendar  quarter  and  submit  it  to  AFSC  (SCSNR) 
not  later  than  45  calendar  days  after  the  end  of  the 
quarter.  The  report  is  to  be  divided  into  three  parts: 

Part  I.  Reliability  Program,  which  includes  certain 
basic  information  such  as  significant  dates  and 
quantitative  requirements. 

Part  II,  Reliability  Status,  which  includes  the 
minimum  acceptable,  predicted,  and  current  status 
values  for  reliability,  maintainability,  and 
availability  (at  system  level  and  for  subsystems). 

Part  III,  Reliability  Problems,  which  includes  a 
narrative  statement  oi  any  existing  problems. 

The  quarterly  reliability  status  report  must  be  pre¬ 
pared  according  to  detailed  instructions  attached  to 
AFSCR  60-1.  For  information  purposes,  the  "Instructions" 
attachment  of  AFSCR  80-1  and  a  sample  reliability  status 
report  form  (AFSC  Form  144)  are  reproduced  here  as  Figure 
3-2  and  Figure  3-3#  respectively. 


3. 1.5. 5  Problems  and  Special  Considerations 
Relating  to  Customer  Internal 
Reliability  Reports 


3.1. 5.5.1  Determination  of  Status  Values 
for  Numerical  Reliability 


Past  experience  has  shown  that,  for  development  pro¬ 
grams,  the  determination  of  the  status  of  numerical  relia¬ 
bility  la  usually  quite  difficult.  The  reason  for  such 
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FIGURE  3-2 

AFSCR  80-1  INSTRUCTIONS  FOR  PREPARATION 
OF  AIR  FORCE  INTERNAL  RELIABILITY  STATUS  REPORTS 

IMTRUCTIMtt  POt  HSPARATXOM  Of  QUARTSLY  RB.XAIXLXTY  STATU*  RSPMTS 

(RCSl  APSC-R44) 


fart  X,  HilUkittty  Pregraa."  Coupleto  itw  1  lktN|k  I),  pert  X  of  AIM  dm  144* 
lteoa  ml  Mate*  below  «r«  eolf  explanatory  i 

leas  •  •  Inter  avaraga  nuaber  af  ua*aMtka  af  after  t  gar  Mack  aaariaf  by  aaa* 
traetar  luring  the  reporting  parte* ,  e.g.,  11  the  aaatraatar  eaaigoe*  X  aaglMara  lav 
1  aenth  an*  only  1  engineer  Car  the  ether  S  aentha,  the  entry  la  tkU  ltaa  wawU  be 

1.44. 

Xtaa  11  -  Intar  numerical  figure  an*  unite  aw  ah  aa  "100  baura  MV  <Haaa  Tlaa 
Between  Pal lure a),"  111  reliability  baa  a*  aa  alaalaa  tlaa  al  10  baura  with  00t  aaa • 
file  nee."  ar  "effaetlvoneea  (uptime),"  ate. 

Xtaa  12  -  inter  nuaerleal  figure  aa*  aana train ta  auah  aa  "1  baur  NRI  Glean  Via* 

Te  Repair)  (aetlve  tlaa  anly)N  ar  "12  baura  aaaa  *awn  tlaa  laaluilng  )  baura  travel 

te  alto,"  ate. 

fart  XX.  "Reliability  Statue."  Ceaplate  part  XX  af  AIM  Para  144  aa  lallawai 
Oaluan  A  •  Inter  eubayataa  Aaalgnatlana. 

Cbaak  apymprlate  boa  abawa  aaluaaa  I.  0.  aa*  0  ta  la*leata  wbatbar  antrlaa  pa** 
tain  te  MTBP,  Paraant.  ar  bath. 

Cbaak  appropriate  ban  above  aaluana  I.  f ,  aa*  0  ta  la*  lea  to  mother  antrlaa  par- 
Min  M  KRR,  Man  Dawn  Tlaa,  or  both. 

OoIwbm  I,  I.  an*  M  -  The an  figure  antrlaa  aheul*  be  either  the  eeatraetually  re¬ 
quire*  aaaa  or  the  non  traetar' a  alleeatlea  to  equipment  or  aubayataa.  aa*  Maul*  bo 
Montlfla*  by  a  footnote  at  bettea  of  the  paga.  Xf  not  appllMbla,  aator  "MA." 

Oolwana  C,  »,  an*  X  -  Thoaa  pra*latlan  flguraa  aheul*  be  In  the  aaaa  unite  aa 
"Mlnlaua  Aaaoptabla"  on*  will  be  aaauaa*  ta  be  aaatraatar 'a  neat  raaoat  pro*Utton 
ualaaa  Mantilla*  aa*  *ata*  ta  another  par  la*. 

Coluana  D,  0,  aa*  J  -  fhns*  atatua  flguraa  art  the  "Reliability"  atatue  of  tfcla 
aye  tan,  aubayataa,  ar  equipment  at  the  S&aa  of  thla  report.  They  aheul*  bo  la  the  aaaa 
unite  aa  the  "Minimum  Aaaapubla"/flguraa. 

Part  XXX,  "Slfnlflaant  RalUblllty  Preblana."  (Reference  ltaa  U,  part  X,  AVSC  Pam 
144.)  prepare  on  •-  by  lOb-laah  boa*  paper  (allow  2/44 laab  bla*lng  aargU  at  toft  of 
pose)  In  narrative  atyto.  Inter  ayataa  or  aajar  aubayataa  title  aa*  nuabar  aa  wall 
aa  the  no  af  «au  of  the  report  at  top  of  aa*  papa.  Vaa  aba  fellawlat  out  lino  boa*- 
lapai 

A.  Area  or  SUbayatea  or  Iquipaaat. 

I.  Irlef  Itataaaat  of  Irablan. 

0.  Pmpata*  baealuttoa. 

>•  Aatloa  Apoaay. 

>.  Ourroat  Mama, 
f.  Ian  ubi. 
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FIGURE  3-3 

SAMPLE  FORM  FOR  QUARTERLY  RELIABILITY  STATUS  REPORT 
(As  per  AFSCR  80-1  "Instructions"  Attachment — See  Figure  V2} 
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FIGURE  3-3  (Continued) 

SAMPLE  FORM  FOR  QUARTERLY  RELIABILITY  STATUS  REPORT 
(As  per  AFSCR  80-1  "Instructions11  Attachment  —  See  Figure  3-2 
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FIGURE  3-3  (Continued) 

SAMPLE  FORM  FOR  QUARTERLY  RELIABILITY  STATUS  REPORT 
(As  per  AFSCR  80-1  "Instructions"  Attachment  --  See  Figure  3-2 
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difficulty  la  the  sparsity  of  test  data  during  early  phases 
of  system  development.  The  most  effective  means  of  alleviat¬ 
ing  this  problem  is  by  striot  enforcement  of  the  contract 
requirements  whioh  relate  to  the  prooess  of  failure  report¬ 
ing  and  recording  of  operating  time. 
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3.2  Reliability  Program  Controls 


3.2.1  Definition 


Program  controls ,  as  discussed  In  this  section,  pertain 
to  those  .devices  or  tools  which  can  be  employed  by  System 
Program  Offices  and  SPO  Reliability  Coordinators  to  follow 
the  progress  of  contracted  reliability  programs  —  that  Is,  to 
compare  actual  achievements  with  desired  or  planned  attain¬ 
ments.  Surveillance  and  review  activities  which  may  be 
considered  as  actions  for  "controlling"  the  over-all  adequacy 
of  contractors'  reliability  programs  are  considered  in  section 
4. 


3.2.2  Air  Force  Policy 


Air  Force  policy  relating  to  establishment  of  program 
controls  is  summarized  in  Table  3-9. 


TABLE  3-9 

SUMMARY  OF  AIR  FORCE  POLICY  RELATING  TO  PROGRAM  CONTROLS 


Air  Force 
Document 

Statement  of  Policy 

APR  80-5 

(1)  Achieving  reliability  is  a  growth 

process.  The  collection,  analysis, 
and  feedback  of  information  to 
both  the  Air  Force  and  industry 
are  fundamental  to  the  control  of 
the  process, - Par*  3-c 

(2)  Management  control  of  reliability 

effort  is  necessary  throughout  a 
system  program. - Par.  3-f 

AFBSDR  30-5 

28  Dec.  1962 

(l)  Procedures  for  program  control 
will  be  developed  and  applied  to 
the  management  activity  at  in¬ 
dividual  systems  management  level. 

Par.  4.h/  Par.  5*a 

3.2-1 


3.2.3 


Responsibilities  for 
Program  Controls 


The  responsibilities  of  SPO  Reliability  Coordinators 
and  associated  agencies  relating  to  program  controls  are 
summarized  in  Table  3-10. 


TABLE  3-10 


SUMMARY  OF  RESPONSIBILITIES  RELATING  TO  PROGRAM  CONTROLS 


Air  Force 
Document 

Level  to 

Which 

Assigned 

Statement  of  Responsibility 

AFSCR-80-1 

14  Dec.  1962 

AFSC  Divisions 
and  Centers 

Maintain  complete,  factual  and 
timely  information  regarding 
the  status  of  reliability 
programs  as  well  as  associated 
program  areas.  Par.  6.f 

System 

Program 

Offices 

(SPO's) 

Maintain  complete,  factual  and 
timely  information  regarding 
the  status  of  reliability 
estimates  and  contractor  re¬ 
liability  programs  as  well  as 
associated  and  program  problem 
areas  Par.  j.h 

AFBSDR  80-5 
28  Dec.  1962 

SPO 

Reliability 

Coordinators 

Maintain  a  current  status  of 
reliability  program  activities, 
achievements,  and  predictions. 

Par.  6.1o.  (3) 

3.2.4  Implementation  of  Program 
Controls 


A  variety  of  devices  in  common  usage,  including  periodic 
status  reports  (see  3.1.1)  and  reliability  growth  curves 
(see  Figure  3-6  in  3. 2. 4. 2. 3),  may  properly  be  classed  as 
program  controls  since  they,  in  some  manner,  permit  comparison 
between  aotual  and  planned  rates  of  progress.  However,  the 


task  of  administering  a  typical  reliability  program  is  of 
such  complexity  that  status  reports  and  simple  growth 
charts  usually  do  not  entirely  satisfy  the  need  for  program 
controls.  As  reliability  programs  become  more  comprehensive, 
a  basic  challenge  posed  to  Air  Force  management  is  the  hand¬ 
ling,  coordinating,  and  display  of  vast  amounts  of  informa¬ 
tion  so  as  to  establish  a  satisfactory  basis  for  program 
decisions.  The  prime  objective  of  a  program  control  device 
is  to  identify  those  events  and  activities  which  are  the 
keynotes  of  program  success. 


As  a  result  of  increasing  attention  devoted  to  develop¬ 
ment  of  program  control  devices  in  recent  years,  several 
systems  are  now  in  existence.  These  systems  differ  greatly 
in  degree  of  sophistication.  For  example,  schedules,  mile¬ 
stone  charts,  or  checklists  have  been  used  in  some  specific 
applications  to  depict  planning  and  accomplishment  of 
program  activities.  In  other  Instances,  more  sensitive 
devices  are  being  continually  developed  and  refined  (e.g., 
the  PERT  and  RMI  systems  discussed  in  following  sections). 

Any  specific  control  device  will  be  characterized  by 
certain  advantages  and,  at  the  same  time,  certain  limitations. 
For  example,  a  milestone  chart  identifies  activities  and 
events  and  Indicates  intended  (and/or  actual)  dates  of 
accomplishment,  but  it  does  not  depict  existing  constraints 
affecting  accomplishment.  Likewise,  a  network  diagram  as 
used  in  the  PERT  system  shows  constraints  (l.e.,  activities 
to  be  accomplished)  between  events,  but  does  not  provide 
a  visual  indication  of  the  "quality"  with  which  events  are 
performed. 


In  some  cases,  the  SPO  Reliability  Coordinator  will 
be  associated  with  a  program  wherein  a  specific  control 
system  or  device  is  contractually  or  otherwise  officially 
prescribed  for  use  throughout'  the  program.  The  primary 
function  of  the  Reliability  Coordinator,  then,  is  to  comply 
with  the  over-all  program  procedure.  He  must  become  familiar 
with  the  details  of  the  control  system,  and  fully  exploit 
it  in  the  administration  of  the  reliability  program. 


Certain  significant  control  systems  which  have  been 
utilized  with  success  in  past  programs  are  discussed  in 

3. 2.4.1  (Formal  Program  Controls)  for  purposes  of  illus¬ 
tration  and  information.  Other  control  forms  which  are 
less  sophisticated  but  whloh  could  be  used  by  a  SPO 
Reliability  Coordinator  in  his  own  applications  are  dis¬ 
cussed  in  3. 2.4. 2  (Simplified  Control  Devices). 


3.2.4. 1  Formal  Program  Controls 


3.2.4. 1.1  Program  Evaluation  and  Review 
Technique  (PERT) 


The  Program  Evaluation  and  Review  Technique  familiarly 
known  as  "PERT"  was  originally  developed  by  the  Navy  Special 
Projects  Office  and  has  since  received  the  endorsement  of 
both  the  Department  of  Defense  and  the  National  Aeronautics 
and  Space  Administration.  While  its  application  is  not 
necessarily  confined  to  reliability  activities,  its  in¬ 
creasing  use  for  administering  reliability  programs  warrants 
brief  consideration  herein.  [Publications  listed  in  the 
bibliography  for  3.2  will  provide  fuller  instructions  as 
to  the  details  of  PERT} 


It  should  be  realized  that  the  basic  PERT  system  is 
not  a  device  for  measuring  the  reliability  of  a  product. 
Rather,  it  is  more  accurately  described  as  a  means  for 
coordinating  the  many  disciplines  involved  in  reliability 
technology. 


The  essential  terms  associated  with  the  PERT  system 

ares 

(1)  Event  —  An  ambiguous  point  in  time  in  the  life 

of  a  project. 

(2)  Activity  --  A  technological  operation  which  con¬ 

sumes  time,  money,  and  manpower.  Each 
activity  is  characterized  by  a  specific 
initial  event  and  terminal  event. 

(3)  Network  —  A  visual  presentation  of  events  and 

activities  which  depicts  interdependencies 


3.2. 4. 1.1.1  PERT  Networks 


An  example  of  a  PERT  network  Is  shown  in  Figure  3-4. 
Note  that  events  are  Joined  together  in  a  manner  which 
illustrates  constraints.  The  method  utilized  in  develop¬ 
ing  any  PERT  network  is  to  combine  various  technologies 
such  as  debign,  testing,  drafting,  reliability,  etc. 
into  a  single  system.  However,  it  is  possible  to  select 
any  one  area  of  endeavor,  such  as  reliability,  and  con¬ 
fine  the  network  to  its  associated  events  and  activities. 
Such  a  network  would  be  of  considerable  value  to  a 
reliability  manager  in  administering  his  program. 


FIGURE  3-4 


EXAMPLE  OF  A  PERT  NETWORK 


Possibly  the  most  difiieult  task  associated  with 
a  PERT  program  Is  the  Initial  selection  of  significant 
program  events.  Reliability  program  specifications  are 
usually  general  In  nature.  Hence,  It  becomes  necessary 
to  translate  these  generalities  Into  carefully  defined 
occurrences  which  can  be  associated  with  specifically 
defined  hardware  Items.  In  selecting  events  which  are 
to  be  controlled,  there  are  certain  factors  which  must 
be  considered.  The  more  significant  of  these  Include 
the  following : 

(1)  Events  must  represent  Instantaneous  points 
In  time. 

(2)  Events  must  be  Identifiable  as  the  respon¬ 
sibility  of  a  particular  working  group 
within  the  contractor's  organization. 

(3)  The  activity  Joining  two  distinct  events 
must  be  capable  of  definition. 

(4)  The  time,  cost,  and  manpower  expenditure  for 
activities  Joining  two  events  must  be  capable 
of  estimation. 


Once  a  network  has  been  developed,  there  are  a  variety 
of  subsequent  procedures  which  could  be  followed.  These 
include  s 

(1)  Estimating  the  time  required  to  complete  each 
activity. 

(2)  Estimating  the  cost  to  accomplish  each  activity. 

(3)  Estimating  the  resources  (in  terms  of  materials 
or  manpower)  required  to  accomplish  each  activity. 


In  some  variations  of  the  basic  PERT  system,  multiple 
estimates  are  made.  For  example.  Instead  of  formulating 
a  single  estimate  of  time  to  accomplish  a  specific  activity, 
it  Is  possible  to  supplement  an  "expected"  time  with  an 
optimistic  estimate  and  a  pessimistic  estimate.  In  so  doing, 
statistical  processes  are  employed  to  determine  the  pro¬ 
bability  of  aocospllshlng  an  event  within  a  given  time. 


3. 2. 4.1.1. 2  Advantages  of  PERT 


i 


The  greatest  single  advantage  of  a  management  system 
such  as  PERT  is  that  it  forces  a  contractor  to  define  the 
elements  of  his  reliability  program.  In  many  complex  relia¬ 
bility  efforts,  this  could  represent  the  most  difficult  task 
of  all.  Other  primary  advantages  of  PERT  are  that  it  can  be 
used  in: 

(1)  Establishing  schedules  and  cost  budgets. 

(2)  Forecasting  total  costs  and  time. 

(3)  Illustrating  time  and  cost  status. 

(4)  Analyzing  manpower  requirements. 

(5)  Determining  the  effects  of  simulated  program 
changes . 

(6)  Providing  basis  for  time/cost  trade-offs. 

(7)  Assisting  in  the  making  of  vital  management 
decisions. 


3. 2. 4. 1.2  Reliability  Maturity  Index  (RMI) 


The  Reliability  Maturity  Index  Is  a  management  system 
which  provides  managers  with  the  Information  necessary  to 
monitor,  control,  and  evaluate  the  progress  of  a  reliability 
documentation  program  and  to  determine  its  effect  upon  the 
reliability  of  the  end  product.  The  RMI  system  is  included 
herein  to  Illustrate  its  features  for  recording,  summarizing, 
and  displaying  reliability  program  information. 

Hie  RMI  system,  which  has  been  pilot-tested  on  the 
Polaris  A3  configuration,  is  composed  of  two  separate  but 
equally  Integral  categories:  the  Schedule  Compliance  Evalua¬ 
tion  (SCE)  and  the  Technical  Quality  Evaluation  (7QE) .  Hie 
SCE  provides  the  planning,  scheduling,  reporting,  and  monitor¬ 
ing  function  of  the  RMI,  and  the  TQE  provides  an  independent 
technical  quality  audit  of  each  reliability  event  document. 


3-2-7 


Specific  advantages  of  RMI  as  a  management  infor¬ 
mation  system  are: 

(1)  Means  are  provided  for  visualizing  the  required 
elements  of  an  effective  operational  reliability 
program  and  progress  (or  non-progress)  in  per¬ 
formance  of  these  elements. 

(2)  Goals  are  established  for  technical  documenta¬ 
tion  and  performance  reporting. 

(3)  The  approach  employed  is  flexible  and  adaptable 
to  the  needs  of  different  subsystems. 

(4)  New  paperwork  is  kept  to  a  minimum  through  use 
of  existing  forms  and  communication  channels; 
required  documentation  is  compatible  with  PERT. 

(5)  A  definitive,  consistent  communication  pro¬ 
cedure  is  established  in  respect  to  reliability 
documentation. 


3. 2. 4. 2  Simplified  Control  Devices 


The  PERT  and  RMI  systems  discussed  in  3. 2. 4.1  are 
examples  of  program  control  systems  which  represent 
relatively  large-scale  operations.  In  addition  to  these  . 
highly  sophisticated  techniques,  there  are  several  other 
control  devices  which  have  been  successfully  applied  in 
specific  situations.  These  include  component  reliability 
status  documents,  milestone  charts,  various  plots  depicting 
program  progress  (e.g.,  reliability  growth  curves),  and 
plots  of  program  expenditures. 


3. 2. 4. 2.1  Component  Reliability 
Status  Document 


This  type  of  document  was  prescribed  within  the  General 
Work  Specifications  for  Atlas  Standard  Space  Launch  Vehicle 
Reliability  Assurance  and  -  - 


e  and  Quality  Assurance  Requirements 
optional  control  device  which  might  be 


Reliability  Ai 

(August  1902 j  as  an  _  _ _  _ „  _ 

employed  by  the  contractor.  It  is,  however,  also  adaptable 
to  use  by  the  customer  for  program  control  purposes. 
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For  the  purposes  of  the  component  reliability  status 
document,  a  component  is  defined  as  an  assembly  of  parts 
into  a  functional  device  which  generally  is  repairable 
by  replacing  a  failed  part  and  which  ordinarily  does  not 
perform  a  useful  task  by  itself.  For  each  component  under 
a  particular  program,  the  document  should  Include  in¬ 
formation  regarding  use  in  each  system  and  subsystem  con¬ 
figuration  and  design  or  production  status.  The  follow¬ 
ing  information  is  prescribed  for  inclusion  in  the  docu¬ 
ments 

(1)  Purpose  or  application  of  component  in 
configuration. 

(2)  Effect  of  failure  of  component  on  system. 

(3)  Summary  of  design  review  analyses. 

(4)  Reliability  estimates  and  growth  curves. 

(5)  Development  schedules  and  development 
progress. 

(6)  Approval  status. 

(7)  Development,  qualification,  acceptance,  and 
demonstration  test  summary  and  results. 

(8)  Reliability  problems  and  proposed  solutions. 

(9)  Reliability  as  designed,  developed,  and  tested 
(comparison  of  estimates  versus  demonstrations). 


The  status  document  should  be  continually  updated  to  serve 
as  a  means  of  comparing  actual  achievements  and  progress 
against  goals.. 


3. 2, 4.2. 2  Milestone  Charts 


Milestone  charts  are  devices  which  enable  identification 
and  scheduling  of  significant  or  key  events  in  a  reliability 
program  and  appropriate  notation  of  the  completion  of  such 
events.  These  charts  are  often  initially  developed  as  part 
of  a  contractor's  reliability  program  plan. 
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Figure  3-5  shows  a  typical  milestone  chart  for  tasks 
to  be  accomplished  in  a  reliability  assurance  program. 

While  this  chart  was  intended  primarily  to  illustrate  a 
situation  involving  a  small  company  (and  hence  a  relatively 
small  program),  the  technique  is  applicable  to  programs 
of  any  size. 


The  same  format  as  illustrated  by  Figure  3-5  could 
be  utilized  to  establish  a  control  device  for  significant 
documents  to  be  submitted  by  the  contractor.  Hence,  a 
listing  of  those  demonstration  plans,  environment  reports, 
periodic  status  reports,  failure  report  summaries,  or  other 
documents  vital  to  the  reliability  program  could  be  made 
the  subject  of  a  special  milestone  chart  with  scheduled 
and  completion  dates  appropriately  depicted. 


3. 2. 4. 2. 3  Plots  Depicting  Program  Progress 


A  plot  of  achieved  numerical  reliability  as  a  function 
of  time  could  be  superimposed  on  a  plot  of  planned  reliability 
growth  to  serve  as  an  effective  program  control  device.  A 
display  such  as  that  shown  in  Figure  3- 6  would  provide  an 
excellent  visualization  of  program  status,  provided,  of  course, 
that  the  data  were  valid. 


Early  in  a  development  program,  there  may  be  little 
observed  data  to  serve  as  the  basis  for  plotting  program 
progress.  As  the  program  matures,  however,  flight  test 
or  field  evaluation  data  will  be  available  in  greater  amount 
and  can  be  used  effectively  in  depicting  progress.  One 
common  plotting  technique  is  illustrated  by  Figure  2-11  in 
2. 1.1. 4. 8, which  shows  the  relationship  of  successful  missions 
to  total  trial  missions  for  space  vehicle  launchings.  In 
that  figure,  both  a  cumulative  plot  of  the  success-to-trial 
ratio  and  a  running  plot  of  the  ratio  for  the  previous  20 
trials  are  presented. 


3. 2. 4. 2. 4  Plots  of  Program  Expenditures 


The  SPO  Reliability  Coordinator  is  concerned  with  control 
of  program  costs  as  well  as  program  progress.  A  plot  of 
cumulative  expenditures  for  reliability  on  a  calendar  time 
basis  can  be  employed  for  visual  comparison  of  planned  versus 
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♦Chart  taken  from  Reliability  —  Small  Companies  — 
Space  Subsystems,  M.  D.  Johnson,  Proceedings,  Ninth 
Satlonal  Symposium  on  Reliability  and  Quality  Control 
January  1963. 


Reliability 


FIGURE  3-6 

COMPARISON  OF  PLANNED  AND  ACHIEVED 
RELIABILITY  GROWTH 
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actual  expenditures  for  a  reliability  program.  Figure 
3-7  Illustrates  one  method  of  plotting  such  Information, 


3.2.5  Problems  and  Special  Considerations 
Relating  to  Program  Controls 


3. 2. 5.1  Responsibility  for  Establishment 
and  Maintenance  of  Controls 


Since  program  controls  are  of  equal  Interest  to  both 
the  customer  and  contractor  management.  It  Is  logical  to 
describe  the  task;  of  establishing  and  maintaining  such 
devices  as  a  Joint  responsibility  of  both  parties.  In 
some  programs.  It  has  been  the  specified  responsibility 
of  the  contractor  to  establish  and  maintain  particular 
program  control  devices  (e.g.,  PERT  networks,  milestone 
charts,  etc.).  In  other  programs,  such  devices  have  been 
utilized  by  the  customer  to  provide  visual  displays  of 
program  progress  at  his  own  facility.  Either  approach 
can  be  equally  effective,  providing  both  the  contractor 
and  the  customer  have  sufficient  access  to  the  required 
Information. 


3.2. 5. 2  Selection  of  Control  Technique 


Because  of  the  many  program  control  devices  developed 
within  recent  years,  confusion  may  exist  as  to  the  specific 
technique  or  techniques  which  would  most  advantageously 
serve  a  given  situation.  The  selection  of  a  basic  method 
(such  as  PERT)  is  usually  not  the  decision  of  the  SPO 
Reliability  Coordinator  but  rather  is  the  result  of  planning 
effort  at  higher  levels  of  authority.  Consequently,  the 
major  responsibility  of  the  Reliability  Coordinator  is  to 
provide  effective  contribution  to  and  derive  maximum  benefit 
from  a  required  over-all  control  system  In  administering 
a  reliability  program.  This  is  not  to  Imply  that  the 
Reliability  Coordinator  oannot  initiate  program  oontrols  at 
his  own  level  of  administration.  Rather,  he  should  augment 
any  larger  program  oontrol  system  by  eatabllahlng  and  maintain¬ 
ing  supplementary  oontrol  devices  (milestone  oharts,  program 
expenditure  plots,  eto.). 
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FIGURE  3-7 

COMPARISON  OF  PLANNED  AND  ACTUAL  EXPENDITURES 
FOR  RELIABILITY  PROGRAM 
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3.2. 5. 3  Availability  of  Time 

for  Maintenance  of  Controls 


It  la  not  uncommon  for  an  administrative  group  to 
design  an  elaborate  program  control  device  and  later 
discover  that  Insufficient  time  Is  available  to  maintain 
the  device.  Hence,  In  selecting  control  devices  for  use 
In  a  specific  reliability  program,  the  SPO  Reliability 
Coordinator  should  envision  the  maintenance  requirements 
and  plan  according  to  anticipated  availability  of  time 
for  properly  updating  information  used  in  the  devices. 
The  Reliability  Coordinator  can  minimize  the  expenditure 
of  his  own  time  by  talcing  advantage  of  visual  displays 
of  program  progress  or  other  oontrols  which  the  con¬ 
tractor  is  implementing  and  by  insuring  that  contractor 
submittals  of  information  are  consistent  with  the  format 
of  those  control  devices  maintained  at  the  SPO  faoillty. 


3.2.6  Additional  References 
on  Program  Controls 


Reliability  and  Product  Assurance.  R.  R.  Landers, 

Prentloe-Hall,  1963,  Chapter  19#  "Reliability 
Control"  pp  445-469. 

PERT- PEP  Reliability  Controls  Techniques  Simplified. 

H.  C.  Romig,  Proceedings,  Eighth  National  Symposium 
on  Reliability  and  Quality  Control,  January  1962. 

PERT  Summary  Report.  Phase  I  and  Phase  II. 

Speolal  Projeots  Of floe.  Bureau  of  Naval  Weapons. 
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3.3  Technical  Direction 

of  Reliability  Programs 


3*3.1  Definition 


As  a  program  administration  task,  technical  direction 
comprises  those  activities  necessary  to  assure  Implementa¬ 
tion  of  a  reliability  program  plan  and  utilization  of 
organized  resources  In  actual  operations  to  achieve  planned 
objectives  of  the  program.  Technical  direction  may  Involve 
a  great  variety  of  decisions,  requests,  and  orders  required 
to  Interpret  or  revise  an  originally  established  plan. 
Closely  allied  to  these  decisions  are  analytical  studies  or 
fact-finding  efforts  initiated  by  either  the  contractor  or 
the  customer. 


3.3.2  Air  Force  Policy 


Air  Force  policy  regarding  the  technical  direction  of 
reliability  programs  is  stated  in  Table  3-11 . 


TABLE  3-11 

SUMMARY  OF  AIR  FORCE  POLICY 
RELATING  TO 


TECHNICAL  DIRECTION  OF  RELIABILITY  PROGRAMS 


Air  Force 
Document 

Statement  of  Policy 

AFBSDR  80-5 

28  Dec.  1962 

It  Is  a  management  responsibility  to  provide 
direction  and  control  to  each  effort  to 
achieve  required  reliability  of  systems. 

Per.  5>'o 

3.3.3  Responsibilities  for  Technical 
Direction 


Assigned  responsibilities  of  SPO  Reliability  Coordi¬ 
nators  and  other  associated  levels  of  authority  relating 
to  technical  direction  of  reliability  programs  are  sum¬ 
marised  In  Table  3-12. 
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TABLE  3-  12 

SUMMARY  OP  RESPONSIBILITIES 
RELATING  TO 


TECHNICAL  DIRECTION  OP  RELIABILITY  PROGRAMS 


Air  Force 
Document 

Level 
to  Which 
Assigned 

Statement  of  Responsibility 

Par. 

No. 

APR  80-5 

4  June  1962 

APSC 

Assume  responsibility  for  all  relia¬ 
bility  design  and  development  test¬ 
ing,  provide  technical  guidance  to 
the  program,  and  evaluate  and  vali¬ 
date  the  results  on  all  systems. 

6.e 

AFSCR  80-1 

14  Dec.  1962 

APSC 

Division  and 
Center 

Provide  technical  assistance  to  the 
CMR's  in  execution  of  their  relia¬ 
bility  responsibilities. 

6.g 

APBSDR  80-5 
28  Dec.  1962 

SPO 

Reliability 

Coordinator* 

Assume  complete  and  over-all  manage¬ 
ment  responsibilities  for  all  aspects 
of  the  system  reliability  program. 

6.b.(3) 

SE/TD 

Contractors 

Be  responsible  for  systems  engineer¬ 
ing  and  teohnioal  direction  of  the 
associate  oontraotor's  reliability 
program. 

6.c.(l) 

3.3*^  Implementation  of  Teohnioal 
Direction 


The  effective  execution  of  the  teohnioal  dlreotlon 
task  requires  that  the  SPO  Reliability  Coordinator: 

(1)  Be  oognizant  of  those  typloal  situations  which 
require  a  deoision  on  his  part. 

(2)  Utilise  available  Information  and  Initiate  sptolal 
studies  to  form  a  basis  for  program  dlreotlon. 


3.3'1**!  Situations  Requiring  Technical 
Erection 

The  following  sections  describe  several  common  occur¬ 
rences  in  contracted  reliability  programs  which  require 
technical  direction  actions  by  the  SPO  Reliability  Co¬ 
ordinator. 


3. 3. 4. 1.1  Changes  in  See 


The  normal  course  of  a  reliability  program  may  lead 
to  a  point  of  decision  regarding  a  change  in  scope  or  re¬ 
orientation  of  effort.  Many  reasons  might  be  cited  as  to 
the  need  for  changing  the  scope  of  a  reliability  program 
from  that  contractually  defined.  As  one  example,  a  parts 
qualification  program  might  develop  information  that  certain 
parts,  previously  thought  to  be  adequate,  are  wholly  in¬ 
adequate  and  that  major  development  work  will  be  necessary 
to  obtain  parts  that  are  useable.  As  another  example,  a 
reliability  program  plan  may  call  for  a  major  effort  in 
qualification  testing  of  subassemblies,  assemblies,  and 
units  and  a  minor  effort  in  flight  testing;  results  of 
lower-level  qualification  tests,  however,  may  indicate  a 
need  for  greater  emphasis  on  flight -testing. 

3. 3. 4. 1.2  Granting  of  Waivers 


Situations  may  arise  in  any  program  wherein  the  con¬ 
tractor  encounters  an  impossible  or  impractical  require¬ 
ment.  The  SPO  Reliability  Coordinator  must  determine  the 
necessity  and  the  procedures  for  granting  such  a  waiver. 
Any  waiver  should  be  documented  and  can  be  made  only  after 
consideration  of  the  effect  upon  product  reliability  and 
after  higher  authority  approval  if  it  affects  a  directive 
of  a  higher  command  level.  In  addition,  the  possibility  of 
decreased  cost  should  be  considered  whenever  a  waiver  is 
contemplated. 


Usually  waiver  actions  are  initiated  by  a  contractor 
request,  but  the  SPO  Reliability  Coordinator  should  consider 
initiating  them  himself  when  such  actions  are  in  the  best 
interests  of  the  program. 
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3. 3. *1.1.3  Acceptance  or  Rejection 
of  End  Item 


At  the  time  of  delivery  of  an  end  item,  a  decision 
must  he  made  to  accept  or  reject  the  item.  This  decision 
can  only  be  based  on  the  contractual  requirements  established 
In  the  specifications  and  work  statement.  Fulfillment  of 
these  requirements  Is  often  sufficiently  subjective  that 
acceptability  is  open  to  technical  Judgment. 


3.3.^.!.^  Withholding  of  Contraot  Funds 


Payments  to  a  contractor  are  dependent  upon  fulfillment 
of  contractual  requirements.  Henoe,  it  is  the  responsibility 
of  the  System  Program  Of floe  to  determine  the  Air  Force's 
right  to  withhold  funds. 


3. 3. 4.1.5  Approval  of  Contractor  Submittals 


In  every  reliability  program,  there  are  numerous 
dooument  submittals  which  require  customer  approval  (e .g., 
the  contractor's  reliability  program  plan).  The  act  of 
approval  (or  disapproval)  constitutes  a  form  of  teohnloal 
direction  and  warrants  timely  and  efficient  aotion  by  the 
SPO  Reliability  Coordinator. 


3. 3. 4.1.6  Interpretation  and  Amplification 
of  Oontraot  Requirements 


One  of  the  more  common  forms  of  teohnloal  direction 
provided  by  the  SPO  Reliability  Coordinator  is  the  inter* 
pretatlon  and  amplification  of  oontraot  requirements.  Such 
dlreotlon  oould  be  the  result  of  a  epeoifio  request  by  the 
contractor  or  oould  be  motivated  by  the  Reliability  Co¬ 
ordinator.  Some  typloal  aotlons  have  been  dlsoussed  at 
length  in  other  seotiona  (e.g.,  see  8.3.1,  which  dlsousses 
instructions  to  the  oontraotor  for  preparing  program  plans, 
and  3 .1.1 .4.1,  whloh  oonalders  instructions  for  perlodlo 
status  reports). 


3. 3. ^.2  Baal 8  for  Executing  Technical 
Direction 


Technical  questions  may  arise  at  any  point  In  the 
progress  of  a  reliability  program,  and  the  ultimate 
responsibility  for  answering  such  questions  most  often 
rests  with  the  SPO  Reliability  Coordinator.  In  deciding 
such  questions,  some  basis  for  judgment  will  always  pre¬ 
vail.  This  basis  Is  desirably  found  In  existing  data, 
information,  or  experience,  but  more  usually  an  analytical 
approach  Is  required.  Hence,  the  Reliability  Coordinator 
may.  In  some  cases,  find  It  necessary  to  direct  a  special 
study  intended  to  provide  required  answers.  This  study 
could  be  undertaken  by  laboratory  tests,  literature 
searching,  data  collection,  or  other  means. 


3. 3. 4.2.1  Use  of  Trade-Off  Curves 


The  use  of  trade-off  curves  In  the  management  of  a 
reliability  program  is  usually  considered  a  planning 
function,  but  the  concept  of  trade-off  can  be  equally 
valid  during  the  performance  of  technical  direction 
activities.  Further,  as  a  program  progresses,  more 
quantitative  information  will  become  available  for 
establishing  trade-off  curves.  When  so  established, 
these  ourves  provide  an  effective  basis  for  formulating 
technical  direction  decisions. 

Slnoe  trade-off  curves  have  been  used  only  recently 
in  managing  reliability  programs,  quantitative  data  re¬ 
lating  to  reliability  trade-offs  are  not  In  abundance. 
However,  considerable  work  Is  being  devoted  to  the 
development  of  the  concept.  Each  SPO  Reliability  Co¬ 
ordinator  has  the  opportunity  to  foster  this  development 
through  application  of  the  trade-off  principle  to  specif io 
situations. 

The  effectiveness  of  any  system  is  a  funotlon  of  many 
interacting  operational  and  design  faotors,  including 
reliability,  design  adequacy,  oost,  schedule,  weight,  and 
maintainability.  Thus,  reliability  oan  be  traded- off  in 
a  great  number  of  ways.  Trade-off  considerations  for 
reliability  are  discussed  in  detail  in  2. 1.1. 4. 5.  An 
extensive  treatment  of  the  subject  oan  also  be  found  in 
the  IRE  Reliability  Training  Text.  Second  Edition,  Maroh 
19^)» 
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3. 3*4. 2. 2  Use  of  Outside  Assistance 


Because  of  the  range  of  specialties  Involved  In  the 
technical  direction  of  a  large  reliability  program,  pro¬ 
vision  of  technical  assistance  to  the  SPO  Reliability 
Coordinator  may  be  warranted.  A  common  procedure  of  the 
Air  Force  Is  to  contract  with  a  technical  organization 
which  has  demonstrated  capability  in  technical  management 
to  supplement  Air  Force  technical  direction  effort.  The 
services  of  this  organization  may  be  procured  for  a  specific 
program  (to  support  the  entire  effort  or  only  a  portion  of 
the  program)  or  on  a  full-time  basis  covering  all  programs. 

Technical  direction  of  a  large-scale  aerospace  relia¬ 
bility  program  requires  an  experienced,  highly  competent 
reservoir  of  technical  capability  to  support  the  relia¬ 
bility  efforts  within  the  various  program  offices  and  their 
many  contractors.  These  efforts  may  present  problems  which 
require  long  or  continuing  study  or  evaluation  by  special¬ 
ists  In  the  reliability  aspects  of  one  or  more  speolflo 
engineering  areas. 


Die  Aerospaoe  Corporation  Reliability  Support  Group 
Is  organized  on  a  functional  basis  and  includes  seotions 
for: 

(1)  System  analysis  for  boosters,  spaoeoraft, 

AGE,  and  applied  problems. 

(2)  Statistical  and  mathematical  analysis. 

(3)  Environmental  studies. 

(4)  Survelllanoe  and  failure  studies. 

(5)  Reliability  data  retrieval(XSBF) . 

The  detailed  functions  of  eaoh  of  these  seotions  are  as 
follows: 

Ere toms  Analysis  Bootlon 

Within  eaoh  of  the  four  major  aotlvity  areas 
(boosters,  spaoeoraft,  AOS,  and  aoplled 
problems),  provide  asslsttaoe  to  8P0  reliability 
staffs  In  formula  tine  work  statements!  generating. 


guiding,  monitoring,  and  reviewing  contractor 
reliability  programs;  and  reviewing  and  analyzing 
reliability  potentials*  including: 

(1)  Assessment  of  reliability  potential. 

(2)  Review  of  electrical  circuits. 

(3)  Review  of  component  parts  selection. 

(4)  Review  of  packaging  design. 

(5)  Review  of  structural  and  mechanical  design. 

(6)  Review  of  thermal  design. 

(7)  Review  of  materials  and  processes. 

(8)  Review  and  analysis  of  contractor  test 
programs,  test  plans,  test  procedures, 
etc., for  physical  significance,  thorough¬ 
ness,  and  competence. 


Statistical  and  Mathematical  Analysis  Section 

(1)  Analyze  contractor  failure  summaries  and  flight 
data. 

(2)  Establish  acceptance  test  procedures  and  relia¬ 
bility  demonstration  criteria. 

(3)  Develop  reliability  models  for  evaluation  and 
assessment . 

(4)  Generate  statistical  guides  for  equipment  on 
parts  acceptance  and  reliability  demonstration 
and  review  design  of  contractor  reliability 
models. 

(5)  Establish  and  maintain  a  statistical  manual  for 
reliability  analyses. 

(6)  Investigate  mathematics  of  reliability  models. 

(7)  Assist  personnel  with  statistical  and  probability 
problems. 

(8)  Supply  computational  assistance. 
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Environmental  Studies  Section 


(1)  Compile,  review,  and  analyze  on  a  continuing 
basis  all  environmental  data  and  documents 
concerning  the  effects  of  earth  and  space 
environments  on  the  reliability  and  performance 
of  spacecraft,  missiles,  and  earth-  or  lunar- 
based  support  equipment. 

(2)  Assist  SPO's  In  establishing  environmental 
guides  and  specifications  for  missile  and 
space  weapon  systems. 

(3)  Determine  the  environmental  requirements  and 
specifications  for  missile  and  space  weapon 
systems  being  considered  for  development. 

(4)  Assess  contractor  capability  for  environmental 
testing  efforts. 

(3)  Assist  with  state-of-the-art  Information  as 
required  In  the  field  of  environments  or 
testing. 

(6)  Monitor  environmental  researoh  and  development, 
qualification,  and  aooeptance  tests. 


Survelllanoe  and  Failure  Analysis  Seotion 

(1)  Review  and  guide  the  efforts  of  missile  and 
spaoe  system  oontraotors  In  establishing  and 
maintaining  failure  reporting  systems  within 
the  soope  of  applicable  contractual  documents 
suoh  as  MIL-R-27542A,  the  work  statement,  and 
the  program  plan. 

(2)  Monitor  oontraotor  failure  reporting  for 
thoroughness,  timeliness,  aoouraoy,  and 
effectiveness  of  oorreotlve  aotions. 

(3)  Review  and  guide  oontraotor  physical  analysis 
of  failed  Items  for  oause  of  failure  and 
determination  of  oorreotlve  aotions. 

(4)  Develop  and  maintain  a  failure  reporting  and 
analysis  capability  at  a  systems  level. 

(5)  Review  the  final  system  configuration  effec¬ 
tiveness  from  the  flight  test  reports. 


(6)  Review  contractor  quality  efforts  in  quality 
assurance,  quality  control,  and  their  appro¬ 
priate  documentation. 


Reliability  Data  Retrieval  Section  (IDEP) 

(1)  Maintain  and  extend  the  scope  of  the  Inter- 
Service  Data  Exohange  Program  (IDEP)  to 
eligible  Air  Force  missile  and  space  con¬ 
tractors  . 

(2)  Strengthen  and  maintain  liaison  with  the 
Army  and  Navy  IDEP  organizations. 

(3)  Act  as  a  primary  data  source  for  other 
aerospace  organizations  for  current  test 
information  on  electrical,  electronic, 
electromechanical,  mechanical,  hydraulic, 
pneumatic,  and  rocket  engine  components. 

(4)  Assist  in  the  interchange  of  "high  relia¬ 
bility"  specifications  among  contractors. 

(5)  Maintain  data  histories  on  parts  through  IDEP 
contractor  participation. 


3.3.5  Additional  References 
on  Technical  Direction 


Air  Foroe  Manual  25-1#  The  Management  Propose, 
September  1954,  pp.  35-42. 
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3.4  Customer  Support  of  Reliability 
Programs 


3.4.1  Definition 


The  measures  taken  to  establish  and  administer  a 
reliability  program  are  negated  unless  the  customer  In¬ 
cludes  a  plan  for  supporting  that  program.  Resources 
and  services  which  the  Air  Force  renders  in  support  of 
contractor  programs  can  be  classified  as:  (1)  funding, 
(2)  provision  of  facilities,  and  (3)  provision  of  a 
variety  of  guidance  information.  Through  these  support 
media,  the  customer  demonstrates  concern  for  the  relia¬ 
bility  of  the  product  to  be  designed,  developed,  or  pro¬ 
duced  and  assists  the  contractor  In  ways  which  may  or 
may  not  be  covered  contractually. 


3.4.2  Air  Force  Policy 


Air  Foroe  policy  pertaining  to  support  of  the  con¬ 
tractor  In  performance  of  a  reliability  program  Is  stated 
In  Table  3-13- 


TABLE  3-13 

SUMMARY  OF  AIR  FORGE  POLICY 
RELATING  TO 


SUPPORT  07  RELIABILITY  PROGRAMS 


Air  Foroe 
Dooument 

Statement  of  Polloy 

AFSCR  80-1 

Ik  Dec.  1962 

New  procrams  will  include  adequate  funds  for 
reliability  effort  in  design  proposals  and 

A7BSDR  80-8 
26  Dec.  1962 

Adequate  funds  will  be  made  available  in  the 
initial  program  funding  for  ballletlo  missile 
system  programs  te  preside  for  the  implementa¬ 
tion  of  a  oomplete  and  well-balanood  relia¬ 
bility  program.  Par.  5.d 

> 


3.4.3  Responsibilities  for  Program 
Support 


Specific  responsibilities  of  the  System  Program 
Offices  and  the  SPO  Reliability  Coordinators  relating 
to  customer  support  of  a  contractor  are  not  explicitly 
cited  in  Air  Force  directives,  but  will  be  stressed  in 
the  following  section  on  implementation. 


3.4.4  Implementation  of  Program 
Support 


3. 4. 4.1  Program  Funding 


One  of  the  more  tangible  and  important  forms  of 
support  which  the  Air  Force  provides  to  a  reliability 
program  is  the  monetary  resources  necessary  to  execute 
the  activity.  Although  funding  is  mainly  a  planning 
consideration,  it  nevertheless  remains  an  essential 
factor  throughout  all  phases  of  the  program. 

The  question  of  "how  much"  to  allocate  to  the 
reliability  program  Is  not  easily  answered.  Although 
several  theories  have  been  advanced,  the  optimum  alloca- 
tion  of  reliability  funds  still  represents  one  of  the 
more  significant  challenges  which  confronts  the  SPO 
Reliatility  Coordinator,  and  one  which  requires  a  delib¬ 
erate  course  of  action  on  his  part. 

BSD-TDR-62-48  (Analysis  of  Reliability  Management 
In  Defense  Industries,  June  19b2)  discusses  the  question 
of  costs  in  a  detailed  manner,  and  cites  a  study  by 
Eric  Pieruschka  (Optimum  Allocation  of  Funds  for  Relia¬ 
bility  Program  for  Guided  Missiles,  Army  Rocket  an3 
bulded  Missile  Agency,  Mudstone  Arsenal,  Alabama.  January 
1955,  revised  1958).  To  quote  from  BSD-TDR-62-48: 

"Pieruschka  argued  that  to  judge  a  missile  program 
by  the  total  cost  of  all  missiles  produced,  good 
or  bad,  is  misleading,  and  bad  missiles  produced 
are  liabilities  rather  than  assets.  His  analysis 
of  the  total  cost  of  a  missile  program  was  related 
to  the  "good"  missiles  only.  Therefore,  the  over¬ 
all  reliability  exhibits  its  outstanding  influence 
on  the  over-all  eoonomy  of  a  missile  program.  He 
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further  claimed  that  the  most  significant  cost 
factor  for  Judging  the  over-all  economy  of  a 
reliability  program  is  the  ratio  between  the 
total  cost  of  a  missile  program  and  the  cost 
of  the  "good"  missiles  that  will  eventually  hit 
the  target.  Pieruschka  observed  that  the  over¬ 
all  effect  of  a  reliability  program  depends  not 
only  on  the  amount  of  reliability  money  avail¬ 
able,  but  also  on  the  skill  of  the  activities 
concerned  with  the  program. 

His  conclusions  of  this  study  were: 

a.  It  is  highly  uneconomical  to  spend 
too  little  effort  and  money  for 
reliability. 

b.  There  1b  little  if  any,  risk  of 
spending  too  much  for  reliability. 

c .  The  optimum  amount  of  funds  for 
reliability  expenditure  should  be 
higher  than  indicated  by  the  minimum 
expenditure  factor,  because  of  mili¬ 
tary  and  political  necessity. 

d.  Contractors'  low  skill  factor  results 
in  a  low  over-all  reliability  and  a 
higher  total  cost  of  a  missile  pro¬ 
gram  , " 

The  oonoluaion  that  "there  is  little,  if  any,  risk 
of  spending  too  muoh  for  reliability"  is  generally 
aooepted  by  those  in  the  field  of  reliability  engineering. 
However,  a  System  Program  Director  must  have  more  faotual 
Information  available  to  allooate  program  funds  in  an 
appropriate  manner.  He  is  dependent  upon  the  SPO  Relia¬ 
bility  Coordinator  to  provide  him  with  substantial  Justi¬ 
fication  for  his  allocations. 

There  are  a  variety  of  ways  in  whloh  the  SPO  Reliability 
Coordinator  oan  contribute  to  the  task  of  properly  allocat¬ 
ing  program  funds .  These  lnolude  suoh  activities  as  analysis 
of  the  oost  of  "unreliability,"  analysis  of  the  worth  of 
reliability  effort,  and  oolleotlon  of  oost  analysis  data, 
whloh  are  disoussed  in  the  following  ssotlons. 


3. 4. 4. 1.1  Analysis  of  Coat  of  Unreliability 


Quite  often  comprehensive  reliability  effort  Is 
Justified  through  an  emphasis  of  the  costs  of  "unrelia¬ 
bility,"  as  pointed  out  In  BSD-TDR-62-48: 

"There  are  many  direct  and  Indirect  costs  related 
to  unreliability.  The  major  direct  costs  are 
(1)  additional  systems  that  are  required  to  carry 
out  a  given  mission,  (2)  additional  spares  used 
In  support  of  the  systems,  (3)  added  bases  and 
Installations  from  which  to  launch  aircraft  and 
missiles,  and  (4)  additional  maintenance  workload 
caused  by  frequent  failures.  The  indirect  costs 
are  (1)  loss  of  prestige  due  to  failures  of  our 
missile  and  space  systems.  (2)  loss  of  weapon  sys¬ 
tem  effectiveness,  and  (3)  false  security  which 
Jeopardizes  this  country's  defense  posture." 

It  was  determined  in  one  study*  that  yearly  support 
costs  for  observed  equipments  were  as  much  as  12  times 
the  initial  procurement  costs,  as  shown  in  Table  3-14: 


TABLE  3-14 

EQUIPMENT  PROCUREMENT  COST  VERSUS  YEARLY  SUPPORT  COST* 


Equipment 

Cost 

Maintenance 

Manpower 

Cost 

Material 

Cost 

Yearly 

Support 

Cost 

Radar 

$400,000 

$230,000 

$10,000 

$240,000 

Communication 

6,000 

70,000 

1,200 

71,200 

Navigation 

5,000 

29,000 

700 

29,700 

It  is  obvious  from  such  figures  that  a  significant  incentive 
prevails  for  reducing  unreliability. 


*The  Measurement  and  Specification  of  Product  Abilities, 
J.  a.  cafaro  and  h.  d.  voegtien,  Quality  control,  Alarch 

1962,  pp.  20-26. 


3. 4.4. 1.2  Analysis  of  Net  Worth 
of  Reliability  Effort 


Quantitative  measurement  of  the  value  of  Improved 
reliability  may  be  difficult  to  achieve  but  nevertheless 
provides  a  significant  and  plausible  challenge  in  justi¬ 
fying  financial  support  of  reliability  effort.  Many  case 
histories  have  been  cited  wherein  effort  expended  during 
development  has  resulted  in  increased  reliability  with  a 
substantial  savings  to  the  government.  Some  of  these 
examples  include: 

(1)  The  AN/ARN-21  TACAN,  purchased  by  the  Air 
Force  in  1958  (see  Reliability  —  Whose 
Responsibility!  H.  L.  Hoffman,  IRe  Trans¬ 
actions  on  Reliability  and  Quality  Control , 
August  19&1,  PP.  4-8). 

(2)  T-38  Reliability  Support  Program,  funded  by 
the  Air  Force  in  January  1949  (see  BSD-TDR- 
62-48,  pp.  8-13  to  8-17; . 

,  t 

(3)  B-58  Reliability  Program,  established  by  the 
Air  Force  in  1959  (see  BSD-TDR-62-4Q  pp.  8-17). 

In  each  of  the  above  oases,  analysis  showed  the  eoonomlo 
advantage  derived  from  a  separately  funded  reliability 
effort . 


3. 4. 4. 1.3  Collection  of  Cost  Analysis  Data 


The  difficulty  of  assessing  the  value  of  eaoh  acti¬ 
vity  within  a  reliability  program  can  never  be  alleviated 
unless  positive  motion  is  taken  to  compile  oost  informa¬ 
tion  with  respect  to  program  elements.  Henoe,  the  SPO 
Reliability  Coordinator  should  exploit  every  opportunity 
to  attain  the  guidance  standard  in  AF  Speolfloatlon 
Bulletin  NO.  506,  ll  May  1959,  which  states: 

"The  program  should  be  so  well  defined  and  broken 
down  to  euoh  detail  that  the  people,  skills, 
facilities,  and  elapsed  tine  requirements  for 
eaoh  specified  activity  can  be  estimated  with 
sufficient  effectiveness  to  establish  program 
oost." 


3. 4. 4. 2  Provision  of  Facilities  and  Equipment 


It  is  often  necessary  for  the  Air  Force  to  support 
a  contractor's  reliability  effort  through  provision  of 
special  facilities  and/or  equipment.  Facilities  may  be 
provided  for  use  only  in  a  particular  program  or  may  be 
of  general  applicability,  extending  over  many  programs 
and  being  available,  through  special  arrangements,  to 
more  than  one  contractor. 

An  obvious  example  of  provided  facilities  is  the 
vast  complex  at  Cape  Canaveral.  It  is  in  the  national 
Interest  to  provide  such  a  site  where  many  contractors 
can  perform  part  of  their  work  using  equipment  avails 
able  to  all  by  arrangement.  Other  facilities  such  as 
test  equipment  are  made  available  to  contractors  depend¬ 
ing  upon  their  need  and  the  need  of  the  program.  Mili¬ 
tary  vehicles  are  made  available  for  tests  when  appro¬ 
priate  to  the  national  Interest.  Such  use  of  government 
facilities  and  equipment  avoids  needless  duplication. 

The  SPO  Reliability  Coordinator's  responsibility 
with  respect  to  facilities  support  is  threefold: 

(1)  He  should  be  congizant  of  the  type,  location, 
and  availability  of  those  government- owned 
facilities  which  could  be  used  in  support  of 
his  program. 

(2)  He  should  apprise  the  contractor  of  these 
facilities  and  encourage  him  to  initiate 
requests  for  their  use. 

(3)  He  should  act  on  contractor  requests  for 

use  of  government- owned  facilities  and  equip¬ 
ment  in  a  manner  consistent  with  the  best 
Interests  of  the  program. 


3.4  .4.3  Provision  of  Ouldanoe  Information 


Examples  of  types  of  guidance  information  which  are 
available  to  contractors  as  a  part  of  oustomer  support  of 
reliability  programs  are  described  in  following  sections. 


3.4. 4.3.1  Military  Standards.  Specifications. 


Bulletins,  and  Exhibits 


Certain  of  the  published  documents  within  this  general 
category  may  well  be  referenced  In  the  contract  for  a 
particular  program.  However,  there  may  be  others  which 
are  not  referenced  contractually  but  which  would  neverthe¬ 
less  provide  substantial  assistance  to  the  contractor 
(e.g..  Air  Force  Specification  Bulletin  No.  510,  Guides 
for  Reliability  Organization) .  The  SPO  Reliability  Co¬ 
ordinator  should  acquire  knowledge  of  government  documents 
which  are  issued  for  purposes  of  guidance  In  contracted 
programs  and  inform  the  contractor  of  the  availability  of 
documents  appropriate  to  his  program. 


3. 4. 4. 3. 2  Educational  Films 


A  variety  of  educational  films  dealing  with  relia¬ 
bility  engineering  are  currently  available  through  govern¬ 
ment  agencies.  These  films  cover  a  wide  range  of  subjects 
and  are  designed  to  reach  personnel  at  various  levels  of 
technical  proficiency.  The  SPO  Reliability  Coordinator 
should  apprise  the  contractor  of  the  availability  of  these 
films  and  encourage  their  use  where  appropriate. 

Included  in  the  available  films  are: 

AGREE  In  Action  --  Produced  by  Aeronautical  Systems 
division,  TlfSAF.  This  film  is  a  report  on  what  Is 
being  done  within  the  present  state-of-the-art  in 
attaining  a  high  level  of  reliability  In  electronic 
equipment  for  a  manned  military  aircraft .  Within 
Its  scope  Is  a  statement  of  policy  concerning  the 
AGREE  task  group,  definitions,  procurement  and 
testing. 

Available  from:  Air  Force  Film  Library  Center 

Air  Photography  Charter  Servloe 
St.  Louis,  Missouri 
Film  No.  TFI-5463 

No  Seoond  Chance  —  Produced  by  Boeing  Airplane  Company. 
An  interesting  story  about  how  one  weapon  system,  an 
interoeptor  missile  on  the  Bomaro  Projeot,  was  made 
reliable.  An  excellent  dissertation  on  the  establish¬ 
ment  of  the  reliability  funotlons  within  a  corporation. 
Time  <  27  minutes . 

Available  from:  Air  Foroe  Film  Library  Center 

Air  Photography  Charter  Servloe 
St.  Louie,  Missouri 
Film  No.  DO-41SKP-456 


The  Price  of  A  Goof  —  Produced  by  Redstone  Arsenal. 
The  story  of  a  failure  of  a  missile  flight  that  is 
traced  from  a  solder  Joint  to  the  abort.  The  film 
illustrates  how  the  improper  joint  managed  to  get 
through  all  the  Inspection  techniques  and  processes, 
check-out  procedures,  and  other  functions  prior  to 
the  firing  of  the  missile.  Time:  18  minutes. 

Available  from:  U.  S.  Army  Ordnance  Dlstrlot 

Dallas  Regional  Office 
Dallas  2,  Texas 

Navy  Reliability  Engineering  Film  Series  (10  Films) 
--  Produced  by  Bureau  of  Naval  Weapons,  Department 
of  the  Navy. 

Available  from:  Bureau  of  Naval  Weapons 

ATTN:  PREN-8 

Reliability  Safety  Section 
Engineering  Branch 
Washington  25,  D.  C. 


3. 4.4. 3. 3  Data  Exchange  Services 


Notable  among  the  data  exchange  services  is  IDEP 
(Interservice  Data  Exchange  Program),  a  government- 
sponsored  tri-service  exchange  of  parts  test  data.  The 
prime  purpose  of  IDEP  is  avoidance  of  duplicate  test 
effort,  but  the  process  of  data  exchange  also  assists  in 
the  selection  and  testing  of  non-standard  parts  and  ulti¬ 
mately  contributes  to  improvement  of  system  performance 
and  reliability. 

IDEP  has  created  a  constantly  expanding  and  up-to- 
date  file  of  documents  relating  to  all  phases  of  parts 
testing.  These  documents  are  primarily  test  reports  on 
"off-the-uhelf"  hardware  likely  to  be  available  among 
participants  in  the  service.  In  addition  to  reports  on 
usual  laboratory- controlled  tests,  the  file  Includes  tabu¬ 
lations  of  parts  data  histories,  general  technical  data 
of  particular  significance  to  parts-uslng  activities,  con¬ 
tractor  specifications  on  hlgh-rellablllty  parts,  and 
aooounts  of  planned  or  in-prooess  parts  testing  activities. 

Participants  in  IDEP  are  government  agencies  and 
prime  contractors  and  major  sub-oontraotors  engaged  in 
parts  testing  activities..  They  are  part  users,  rather 
than  parts  manufacturers  or  vendors. 


The  SPO  Reliability  Coordinator  should  actively  en¬ 
courage  contractor  participation  in  programs  such  as  IDEP. 

In  some  programs,  participation  in  IDEP  has  been  a  con¬ 
tractual  stipulation. 

3-4. 3*4  Information  Services 

The  most  exhaustive  of  the  information  services  currently 
in  existence  is  the  Defense  Documentation  Center  (DDC). 
Operated  by  the  Department  of  Defense.  DDC  provides  a 
central  service  for  the  collection  and  interchange  of 
technical  and  scientific  reports  which  have  direct 
bearing  on  defense  research  and  development  efforts.  The 
scope  of  coverage  provided  by  the  service  is  virtually 
unlimited  and  includes  subject  matter  directly  related  to 
reliability  programs.  Information  concerning  the  avail¬ 
ability  of  reports  and  data  is  provided  through  published 
bibliographies  and  semimonthly  announcement  bulletins  of 
new  acquisitions.  In  addition,  DDC  provides,  on  request, 
bibliographies  on  specially  selected  subjects. 

The  DDC  service  is  available  not  only  to  all  govern¬ 
ment  agencies  but  also  to  contractors  and  subcontractors 
engaged  in  defense  programs. 

The  SPO  Reliability  Coordinator  should  therefore 
encourage  reliability  program  participants  to  utilize  the 
DDC  facilities  to  the  fullest  extent. 


3.4.5  Problems  and  Special  Considerations 
Relating  to  Program  Support 

3.4. 5.1  Stability  of  Contract  Funding 

Efficient  planning  and  subsequent  implementation  of 
reliability  programs  depends  to  a  large  extent  on  the 
stability  of  funding  support.  Fluctuation  of  funds  In  a 
program  can  greatly  hinder  the  task  of  planning. 

There  is  no  ready  solution  to  assurance  of  a  constant 
adequacy  of  funds  since  certain  readjustments  will  usually 
be  a  budgetary  necessity  in  most  programs.  However,  the 
SPO  Reliability  Coordinator  Should  exert  any  effort  possible 


to  minimize  the  occurrence  of  unstablllty  in  funding  sup¬ 
port.  The  Reliability  Coordinator  can  contribute  to  the 
provision  of  stable  and  adequate  funding  support  by: 

(1)  Specifying  elements  of  work  within  the  contract 
in  a  clear,  concise  manner,  thereby  enabling  a 
prospective  contractor  to  estimate  costs 
realistically. 

(2)  Insuring  that  the  Interests  of  the  reliability 
program  are  adequately  represented  and  Justified 
in  apportioning  over-all  program  funds. 

(3)  Assessing  reliability  achievements  (or  potential 
achievements)  from  the  standpoint  of  economic 
advantage . 

(4)  Assuring  that  currently  allotted  funds  are  being 
expended  in  an  efficient  manner,  nils  can  only 
be  accomplished  through  the  maintenance  of  appro¬ 
priate  cost-breakdown  information  for  a  program. 


3.4.6  Additional  References 
on  Program  Support 


Funding  Reliability  Programs ,  E.  F.  Dertinger, 
Troceedlngs,  Ninth  National  Symposium  on  Relia¬ 
bility  and  Quality  Control,  January  1963*  P-  16. 

Minimizing  the  Cost  of  Reliability  Testing, 

H.  TT  Maloney,  Proceedings,  Seventh  National 
Symposium  on  Reliability  and  Quality  Control, 
January  1961,  p.  313 • 

Pie  Dollar  Value  of  Improved  Reliability, 

E.  L.  welker  and  <3.  E.  Bradley,  Proceedings, 
Seventh  National  Symposium  on  Reliability  and 
Quality  Control,  Jenuary  1961,  p.  323. 


Reliability  —  Buy  Now  and  Don't  Pay  Later, 

V.  F.  Stevens,  Proceedings,  Sixth  National 
Symposium  on  Reliability  and  Quality  Control , 
January  i960,  p.  93* 


Rework  Costa  Related  to  Reliability 
V7TT  Kuimln7  *f?olYedIRga ,  glxEnfc 
on  Reliability  and  Quality  Control, 
9-  95. 


Requirements, 
tionai  symposium 
January  i960. 
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Reliability  Versus  the  Coat  of  Failure, 

A.  A.  Raymond,  Proceedings ,  fourth  National 
Symposium  on  Reliability  and  Quality  Control, 
January  1958,  p.  187. 

The  Price  of  Reliability,  A.  L.  Lambert, 
Proceedings,  Fourth  National  Symposium  on 
Reliability  and  Quality  Control,  January  1958, 
p.  189. 

R.  and  D  Reliability  and  Dollars.  E.  T.  Welmers, 
National  Symposium  on  Reliability  and  Qualtly 
Control,  January  1958,  p.  194. 


O 
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4.  SURVEILLANCE  AND  REVIEW  OP  A  RELIABILITY  PROGRAM 


In  a  reliability  program,  the  desired  end  product  Is 
a  system  which  exhibits  (usually  in  the  field)  a  capability 
of  accomplishing  a  mission  with  some  pre-deflned  probability 
of  success.  The  effectiveness  of  the  program  can  only  be 
Judged  by  the  performance  which  the  system  achieves.  Be¬ 
cause  of  the  scope  of  most  reliability  programs,  however, 
the  Air  Force  oannot  afford  to  withhold  evaluation  until  the 
end  product  results.  It  is  essential,  therefore,  that  some 
form  of  evaluation  be  practiced  throughout  the  duration  of 
the  program  —  to  insure  that  performance  Is  In  aocordanoe 
with  established  standards  and  that  the  standards  oontlnue 
to  reflect  the  program  needs. 

The  evaluation  funotlon,  as  discussed  In  section  4, 
involves  two  distinct  areas  of  activity  —  "surveillance" 
(4.1)  and  "review"  (4.2).*  These  activities  are  defined 
as  follows: 

(1)  Surveillance  —  the  observation  of  Individually 
defined  work  elements  of  a  program  at  frequent, 
repetitive  intervals.  Close,  continual  super¬ 
vision  of  the  program  Is  implied. 

(2)  Review  —  a  formal  and  offioial  aotion  whioh 
ooours  periodically  and  whioh  cumulatively 
considers  all  elements  of  the  program. 

Both  survelllanoe  and  review  of  a  reliability  program 
are  dearly  within  the  responsibility  assignment  of  a  SPO 
Reliability  Coordinator.  Eaoh  task  requires  a  three-step 
management  prooess:  development  of  a  plan  (including 
establishment  of  evaluation  standards  and  orlterla),  exe¬ 
cution  of  the  plan,  and  feedback  of  results  to  the  relia¬ 
bility  program. 

The  bases  for  program  evaluation,  In  either  survell¬ 
lanoe  or  review,  oomprlse  both  qualitative  and  quantitative 
orlterla.  Qualitative  orlterla  are  oonoemed  mainly  with 
the  "goodness"  of  the  end  produot  or  program  aooompllsh- 
ments,  and  their  applloatlon  results  In  some  value  statement 

*btner  terminology  suoh  as  "monitoring"  and  "audit"  have 
frequently  been  used  In  praotloe  to  desorlbe  program 
•valuation  functions  whioh  are  equal  or  similar  to  those 
disousaed  in  seotlon  4. 


or  Judgment  of  varying  accuracy  and  objectivity.  Quanti¬ 
tative  criteria  are  concerned  with  the  magnitude  of  product 
capability  or  program  progress,  with  their  application 
results  in  some  numerical  expression  Involving  units  of 
count  or  measurement. 

When  the  effectiveness  of  an  operational  system  is 
Judged,  the  strictly  qualitative  criteria,  usually  far  out¬ 
weigh  the  quantitative  aspects  in  number  and  importance. 
Hence  a  manager  must  often  must  decide  "how  stood"  is  the 
contractor's  performance,  or,  is  he  doing  a  ^satisfactory" 
Job?  On  the  other  hand,  a  program  manager  cannot  operate 
without  facing  some  need  for  quantitative  data.  Such 
questions  as  how  long  will  it  take  to  complete  a  given 
Job,  how  many  working  days  of  how  many  people  will  be  re¬ 
quired,  how  much  of  the  necessary  materials  must  be  used, 
and  how  much  will  the  Job  cost  must  all  be  answered  in 
quantities.  Hence,  quantitative  standards  of  performance 
are  an  essential  requirement  for  program  management  and 
evaluation . 


4-2 


« 


4.1  Program  Survelllanos 


4,1.1  Definition 


As  previously  defined,  surveillance  connotes  oustomer 
activities  associated  with  continual,  close  supervision  of 
the  Individual  elements  of  work  In  a  oontrao tor's  reliabil¬ 
ity  program.  Surveillance  commences  upon  completion  of  the 
establishment  of  the  program  (see  section  2).  and  continues 
throughout  Its  duration. 


4.1.2  Air  Force  Policy 


Air  Foroe  policy  with  respect  to  the  taak  of  surveil¬ 
lance  Is  summarised  In  Table  4-1. 

TABLE  4-1 

SUMMARY  OF  AIR  FORCE  POLICY 
RELATINO  TO  SURVEILLANCE  OF  RELIABILITY  PROORAMS 


Air  Foroe 
Dooument 

Statement  of  Polloy 

AFR  80-3 

U  June  1962 

The  Air  Foroe  will  maintain  aurvelllanoe 
over  the  contractor's  reliability  program, 
reliability  testing,  and  quality  control 
activities.  Par.  U.d 

AFB8X3R  80-3 

28  Dec.  1962 

Speolflo  reliability  programs  will  be 
established  and  monitored  for  all  ballis¬ 
tic  systems  or  portion*  thereof  having 
separate  Air  Foroe  oontraets  for  their 
development/proouresMnt .  par.  5 

4.1,3 


BlIMMttUlttll  Itt 


Tha  speoifio  survelllaaoe  rtiptMlblllttii  of  StO 
Reliability  Pear dlnatars  JEST associated  leaala  of  authority 
art  suMaarlsed  la  tibia  4-1. 
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TABLE  4-2 

SUMMARY  OF  RESPONSIBILITIES 
RELATINO  TO  SURVEILLANCE  OF  RELIABILITY  PROGRAMS 


Air  Force 
Document 

Level 
to  Which 
Assigned 

Statement  of  Responsibility 

Par. 

No. 

APR  80-5, 

4  June  1962 

AFSC 

(1)  Determine  the  adequacy  of  each  Air 

Force  contractor's  reliability  program 
for  achieving  and  demonstrating  Air  Force 
product  reliability  goals.  _  _ 

6.d 

AFSCR  80-1 
i4  Dec.  1962 

AFSC 

DCS/Systems 

(l)  Prescribe  policies  and  procedures  as 
general  guides  to  be  followed  by  AFSC 
systems  divisions  and  centers  In  monitor¬ 
ing  and  reporting  on  the  reliability 
programs  for  the  various  systems,  sub- 
"  systems,  and  equipment  under  development. 

5.  a.  (4) 

AFSC 

DCS/ 

Procurement 

and 

Materiel 

(l)  Prescribe  policies  and  procedures  as 
guidance  for  the  CMR  supporting  the  sys¬ 
tem  program  office  (SPO  in  achievement  of 
contractual  reliability  requirements. 

5-b.(2) 

(2)  Prescribe  policies  and  procedures  for 
maintaining  surveillance  during  all  phases 
of  development  and  production  over  the 
contractor's  quality  control  activities. 

5-b.(3) 

• 

System 

Program 

Offices 

(SPO's) 

(l)  Maintain  ah  adequate  staff  for  moni¬ 
toring  and  guidance  of  the  reliability 
program  for  the  system  with  which  they 
are  concerned. 

T.a 

CMR 

(l)  Maintain  surveillance  during  develop¬ 
ment  and  production  to  assure  contractor's 
compliance  with  the  contractual  reliability 
reauirements . 

8 

BSDR  80-5 

28  Dec.  1962 

SPO 

Reliability 

Coordinators 

(l)  Evaluate  all  aspects  of  the  contrac¬ 
tor's  reliability  program  and  take  action 
as  necessary  to  assure  the  adequacy  of 
this  program. 

6.b.(3) 

SE/TD 

Contractors 

(l)  Evaluate  and  assure  that  data  and 
assumptions  used  by  the  contractor  are 
valid  prior  to  preparation  of  plans  for 
demonstration  of  achieved  reliability. 

6.c.(3) 

(2)  Analyze  the  over-all  performance  of 
the  weapon  system  and  its  components  as 
observed  during  tests  and  compare  achieve¬ 
ments  to  reliability  predictions  and 
established  goals. 

S.c.(lO) 

4.1.4  Implementation  of  Program  Surveillance 


The  task  of  surveillance  Is  implemented  through  a 
three-step  process  --  development  of  s  positive  plan  with 
well-defined  surveillance  standards  and  schedules,  trans¬ 
lation  of  the  surveillance  plan  into  effective  motions, 
and  feed-baok  of  surveillance  results  to  the  contractor 
and  Air  Foroe  authorities. 


4. 1.4.1  Development  of  Surveillance  Plan 


Development  of  a  surveillance  plan  for  application 
to  a  speoiflc  reliability  program  should  enoompass  the 
following  procedures » 

(1)  Determine  and  list  the  elements  of  work  to 
be  surveyed. 

(2)  Determine  and  list  the  standards  whloh  will 
be  used  in  supervising  saoh  element  of  work. 

(3)  Determine  the  frequency  with  which  judgments 
of  program  progress  and  quality  (based  on 
comparison  of  standards  and  accomplishments) 
will  be  made. 


4. 1.4. 1.1  Elements  of  Work  to  Be  Surveyed 


Ideally,  the  speeiflo  elements  of  work  to  be  sur¬ 
veyed  should  have  been  previously  established  by  the 
original  contractual  statement  of  requirements  and/or 
the  oontraotor's  reliability  program  plan.  Renee,  if 
appropriate,  definition  of  the  areas  of  surveillance 
may  simply  Involve  reference  to  those  documents. 

For  purposes  of  illustration,  the  major  elements 
of  work  could  be  described  by  the  following  11s ting  t 

(1)  Oeneral  considerations. 

(2)  Numerical  reliability  requirements. 

(3)  Program  revises. 

(4)  Reliability  interface  with  total  program. 

(3)  Design  reviews. 


* 


(6)  Design  change  and  control, 

(7)  Mathematical  models 

(8)  Integrated  test  program 

(9)  Exchange  of  data. 

(10)  Problem  control  and  corrective  action. 

(11)  Reliability  demonstration. 

(12)  Failure  reporting  system. 

(13)  Reports  submittal. 

The  above  listing  of  reliability  program  elements  is  only 
representative  of  many  that  have  been  developed;  other 
examples  can  be  found  in  standard  specifications,  technical 
papers ,  and  in  Appendix  A  to  this  handbook. 

The  degree  of  detail  to  which  the  elements  are  to  be 
defined  is  subject  to  the  discretion  Of  the  individual  SPO 
Reliability  Coordinator  and  available  manpower.  For  small 
reliability  programs,  a  listing  of  the  major  work  cate¬ 
gories  requiring  surveillance,  such  as  that  above,  may  be 
sufficient.  For  large  or  "comprehensive"  reliability  pro¬ 
grams,  it  is  usually  more  practical  to  expand  the  major 
categories  and  define  more  detailed  levels  of  surveillanoe . 
Examples  of  such  descriptive  listings  of  elements  to  be 
surveyed  can  be  found  in  the  following  sources: 

(1)  Reliability  Program  Check  List,  Aerospace 
Corporation  Publication  No.  1923-1-325  — 

This  check  list,  is  keyed  to  the  activities 
and  tasks  of  MIL-R-27 5^2,  MIL-Q-9858,  MIL- 
STD- 105,  MIL-STD-414,  and  MIL-STD-810.  The 
listing  was  compiled  for  use  in  checking  work 
statements,  proposals,  or  program  plans  for 
new  or  Intended  programs. 

(2)  Quldes  for  Reliability  Organization,  Air  Force 
Specification  SulletinNoT  510,  dated  30  June 
1959  — This  document  incorporates  a  master 
check  list  of  reliability  program  elements  for 
use  on  complex  military  weapon  systems* 


4. 1.2. 1.2  Standards  for  Surveillanoe 


An  outline  of  the  elements  of  a  reliability  program, 
ms  presented  in  the  oon tract  and/or  the  reliability  pro¬ 
gram  p$an,  merely  identifies  the  areas  of  sureeillimee . 
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The  second  step  In  the  development  of  the  surveillance 
plan  Is  the  establishment  of  standards  for  judging  work 
performance.  There  are  two  basic  standards  whloh  glvern 
surveillance  actions: 

(1)  Was  planned  work  accomplished  In  accordance 
with  pre-established  schedules? 

(2)  Was  the  work  performed  with  sufficient  quality? 


4. 1.4. 1.2.1  Surveillance  of  Compliance 
to  Schedule 


Judging  a  contractor's  performance  for  compliance  to 
schedule  presumes  the  existence  of  defined  tasks,  a  sched¬ 
ule  for  completion  of  these  tasks,  and  an  Inherent  ability 
to  determine  positively  whether  the  task  was  completed. 
Consequently,  this  surveillance  function  Is  closely  related 
to  the  speolflc  program  control  devloes  (see  3.2]  which 
are  employed.  The  PERT  and  RMI  oontrol  systems  (see 
3. 2. 4. 1.1  and  3. 2. 4. 1.2)  require  definition  and  scheduling 
of  tasks  or  activities  and  provide  means  for  ready  visual¬ 
isation  of  compliance  to  sohedule.  In  the  RMI  system,  for 
example,  task  completions  are  Identified  by  submittals  of 
speolfloally  titled  documents  generated  in  the  oourse  of 
a  program. 


4. 1.4. 1.2. 2  Surveillance  of  Quality 
of  Performance 


In  Judging  the  quality  of  a  contractor's  performance. 

It  beoomss  neoessary  to  attaoh  a  figure  of  merit  to  eaoh 
element  of  work.  This  figure  of  merit  oould  be  a  numer¬ 
ical  soore  or  a  judgment  of  "satisfactory"  or  "unsatis¬ 
factory."  As  applioable  to  the  task  of  quality  survslllanoe, 
general  Air  Foros  polloy  oan  be  Interpreted  as  follows: 
Pre-established  standards  should  be  utilised  to  the  extent 
that  they  provide  an  effeotlve  vehlole  for  Judgment,  but 
Judgment  should  not  be  based  solely  on  these  standards. 

The  quantity  of  faotors  whloh  must  be  oonsldered  In 
•urvelllanoe  of  reliability  program  elements  suggests 
that  a  oheok  list  apnroaoh  oould  be  utilised.  This 
gMPMoh  would  oommae  selection  of  a  controllable  number 
or  work  elements  which  cumulatively  describe  the  reliability 
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program  (sea  4.1. 4.1.1)  and  compilation  of  quality  standards 
and  check  lists  for  each  element.  For  the  most  part,  appro¬ 
priate  standards  will  be  reflected  by  the  contract  statement 
of  requirements  and  by  the  contractor's  reliability  program 
plan.  However,  supplemental  Instructions  and  directives 
issued  by  the  Air  Force  (specifically  to  the  contractor  or 
as  general  publications  --  e.g..  Air  Force  Specification 
Bulletins  Nos.  506  and  510)  as  well  as  sound  engineering 
practice  may  also  determine  quality  standards. 

One  method  of  setting  forth  surveillance  standards  Is 
to  compile  a  notebook  which  is  specifically  tailored  to 
the  program  of  interest.  The  notebook  should  be  divided 
into  sections  which  represent  the  major  elements  of  work 
to  be  surveyed.  Then,  for  each  major  element,  all  pertinent 
contract  statements,  reliability  program  plan  statements, 
and  supplemental  instructions  and  schedules  would  be  accumu¬ 
lated  in  the  appropriate  section  of  the  notebook.  In  addi¬ 
tion,  check  lists  which  represent  Implied  standards  could 
be  Included. 

The  compilation  of  a  standards  notebook  for  a  program 
must  be  considered  a  dynamic  rather  than  static  task.  In 
any  program,  originally  stipulated  standards  (i.e.,  those 
appearing  in  the  contract)  will  be  continuously  amplified 
and  interpreted.  Hence,  it  is  logical  that  the  initial 
notebook  would  also  be  subjected  to  continual  refinement 
and  expansion.  In  this  sense,  the  notebook  provides  a 
vehicle  for  convenient  filing  and  location  of  standards 
resulting  from  normal  management  of  the  program. 

A  sample  section  of  the  suggested  standards  notebook 
for  one  major  program  element  —  the  failure  reporting 
system  --  is  presented  as  Figure  4-1.  Each  major  element 
of  the  over-all  reliability  program  could  be  similarly 
treated.  Items  1  through  5  represent  official  documenta¬ 
tion  which  is  included  in  a  normal  program.  Item  6,  "Other 
Quality  Standards,"  represents  those  factors  which  result 
from  implication  or  interpretation  of  contract  statements. 

In  essence,  these  factors  are  "check  list"  items  that  are 
developed  by  the  SPO  Reliability  Coordinator  on  the  basis 
of  his  own  experience  or  information  contained  within 
available  literature. 
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FIGURE  4-1 

SAMPLE  SECTION  OF  STANDARDS  NOTEBOOK 
FOR  SURVEILLANCE  OF  RELIABILITY  PROGRAMS 


SURVEILLANCE  STANDARDS  FOR: 

Failure  Reporting  System 
(Program  Work  Category) 


(1)  Applicable  paragraphs.  Statement  of  Work: 

(Reference  paragraphs  and  attach  copies  aa  exhibit) 

(2)  Applicable  paragraphs,  MIL-R-27542: 

(Reference  paragraphs  and  attach  copies  aa  exhibit) 

(3)  Applicable  paragraphs.  Reliability  Program  Plan: 
(Reference  paragraphs  and  attach  copies  as  exhibit) 

(4)  Supplemental  standards:  (e.g.,  Contractor  Document 
No,  XXX.  XYZ  Company  standard  operating  procedure 
for  failure  reporting) 

(5)  Sohadullng  standards:  (e.g..  Monthly  failure  sum¬ 
maries  to  be  submitted  l5th  of  eaoh  month  —  attach 
oopy  of  required  summary  contents  as  exhibit j 

(6)  Other  quality  standards: 

(a)  Are  forms  being  completely  filled  out? 

(Cheok  sample  of  100  submitted  forms,  to 
determine  percentage  completion  for  seleoted 
Items . ) 

(b)  Are  operating  times  being  reoorded  on  failure  forms? 

(o)  From  sampling,  do  all  reported  failures  appear  In 
failure  summary? 

(d)  From  sampling,  what  Is  average  time  required  to 
olose  out  a  failure  report? 

(s)  What  percentage  of  reported  failures  havs  been 
classified  as  to  oauss? 

(f )  Are  failure  reports  being  soreened  for  aoouracy? 

(g)  Are  failure  reports  being  received  from  vendors? 

(h)  Does  oontraotor  know  whloh  reports  are  not  yet 

_ am _ i 


4. 1.4. 1.3  Scheduling  of  Surveillance 


The  final  step  In  the  development  of  a  surveillance 
plan  Is  the  determination  of  the  frequency  of  surveillance 
activities.  One  approach  Is  to  perform  surveillance  on  a 
continuous  basis.  This  approach  is  not  normally  conducive 
to  organized  execution,  particularly  since  the  surveillance 
function  involves  other  administrative  duties.  Hence,  it 
is  more  practical  to  establish  a  working  schedule  for  con¬ 
duct  of  the  function. 

A  surveillance  schedule  need  not  be  complex.  For 
example,  the  thirteen  major  areas  of  consideration  pre¬ 
viously  listed  in  4. 1.4. 1.1  could  be  utilized  as  the  basis 
for  the  schedule  with  each  element  scheduled  on  either  a 
random  or  regular  basis.  The  decisions  necess;  ry  In 
development  of  the  schedule  include: 

(1)  The  frequency  with  which  each  defined  element 
will  be  investigated. 

(2)  The  amount  of  time  to  be  expended  in  each 
investigation. 

These  decisions  are  dependent  on  the  amount  of  time  to  be 
allotted  to  the  surveillance  function. 


4. 1.4. 2  Conduct  of  Surveillance 


The  effectiveness  with  which  any  particular  element 
of  work  is  investigated  will  depend  on  the  degree  of 
completeness  with  which  the  quality  standards  and  schedule 
compliance  standards  have  been  previously  developed.  The 
execution  of  the  surveillance  plan  involves  comparison  of 
these  standards  with  demonstrable  progress,  as  manifest  in 
program  documentation,  observation  of  events,  and  inter¬ 
views  with  appropriate  personnel. 

In  conducting  surveillance,  the  SPO  Reliability  Co¬ 
ordinator  should  coordinate  his  activities  with  the  Air 
Force  Plant  Representative  Office  assigned  to  the  con¬ 
tractor's  facility.  The  APPRO  objective  of  on-site 
inspection  coincides  with  the  survelllanoe  task  of  the 
Reliability  Coordinator.  Since  the  specific  relationship 
between  the  SPO  and  the  AFPRO  may  vary  with  the  Air  Force 
management  policy  for  each  individual  program,  it  is  in¬ 
cumbent  on  the  SPO  Reliability  Coordinator  to  coordinate 
the  execution  of  the  reliability  program  survelllanoe  plan 
with  the  AFPRO  inspection  activities.  (See  par.  3*1 *4.4.2) 
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4. 1.4. 3  Feedback  of  Surveillance  Results 


The  manner  In  which  the  results  of  survelllanoe  are 
used  will  depend  on  the  criticalness  of  the  deficiencies 
found.  Significant  departures  from  quality  or  schedule 
compliance  standards  or  problems  which  cannot  be  resolved 
through  usual  communication  channels  should  be  formally 
reported  to  higher  Air  Force  authority.  In  Instances  of 
less  significant  deficiencies,  the  SPO  Reliability  Co¬ 
ordinator  may  elect  to  advice  the  contractor  in  a  more 
routine  manner. 

The  results  of  surveillance  actions  provide  important 
contributions  to  formal  program  reviews  (see  4.2).  There¬ 
fore,  it  is  paramount  that  the  results  be  oomplled  in 
documentary  form. 


4.1.5  Additional  References 

on  Program  Survelllanoe 


Aspects  of  An  Effective  Reliability  Program 
Implemented  for  An  Advanced  Bombing- Navigation 
System,  F.  V.  Blackburn  and  A.  M.  Le tow.  Pro¬ 
ceedings,  Eighth  National  Symposium  on  Reliability 
and  Quality  Control,  January  1962,  pp.  272-280. 


Scheduling  Supplier  Surveys,  D.  L.  Field,  Pro- 
oeedings,  Eighth  National  Symposium  on  Reliability 
and  Quality  Control,  January  1962,  pp.  431-433. 


Military  System  Reliability,  J.  Spiegel  and 
e.  m.  Bennett, TUB  transactions  on  Reliability 
and  Quality  Control,  Vol.  RQC-10,  No.  2 
(August  1961),  pp.  53-63. 


Talos  Missile  Reliability  Data  System  Contribution 
to  Management,  N.  R.  Wenat,  fourth  National  Symposium 
on  Reliability  and  Quality  Control,  January  1938, 

P.  223. 

Reliability  end  Quality  Control,  January  1937# 
pp.  133-139. 


» 
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4.2  Program  Review 


4.2.1  Definition 


The  review  of  a  reliability  program  differs  from 
surveillance  of  the  program  in  that  the  former  is  more 
formal  and  deliberate  in  nature  and  occurs  at  discrete 
points  in  time,  while  the  latter  is  a  continual,  super¬ 
visory  type  of  action.  Further,  a  program  review  is 
concerned  more  with  the  progress  of  the  over-all  relia¬ 
bility  program  rather  than  its  individual  elements. 

Since  the  review  of  a  reliability  program  could 
employ  a  variety  of  techniques  and  be  governed  by  a 
variety  of  procedures,  the  success  of  this  activity 
is  contingent  upon  the  precise  stipulation  of  organized 
procedural  details. 


4.2.2  Air  Force  Policy 


Air  Force  policy  relating  to  formal  review  of  relia¬ 
bility  programs  is  summarized  in  Table  4-3. 


4.2.3  Responsibilities  for  Program  Review 


Assigned  responsibilities  of  SPO  Reliability  Co¬ 
ordinators  and  other  associated  groups  relating  to 
program  review  are  presented  in  Table  4-4. 


4.2.4  Implementation  of  Program  Review 

Die  review  function  is  discussed  under  two  separate 
headings: 

(1)  Periodic  program  reviews 

(2)  Major  milestone  reviews 
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TABLE  4-3  9 

SUMMARY  0?  AIR  FORCE  POLICY 
RELATING  TO  REVIEW  OF  RELIABILITY  PROGRAMS 


Air  Force 
Document 

Statement  of  Policy 

Par. 

Ho. 

AFR  80-5 

1+  June  1962 

(l)  Management  review  of  reliability  effort  le 
necessary  throughout  a  system  program.  The  review 
must  include  an  evaluation  of  predicted  operational 
reliability  and,  as  applicable,  measurement  of 
achieved  reliability  at  every  step  in  the  program. 

3-f 

(2)  Reliability  monitoring  points  generally  will 
be  established  in  the  following  sequence: 

4.h 

[1] 

'2' 

! 

1 

Detailed  design  study 

Preprototype 

Prototype 

Preproduction  demonstration 

Demonstration  of  service  readiness 

Service  evaluation 

Full-scale  production 

Demonstration  of  major  product  improvement 

However,  this  generalization  is  not  intended  to 
delineate  the  oomplete  or  ideal  system  life  oycle, 
but  to  emphasize  the  typloal  points  at  which  the 
program  should  be  monitored. 

AFSCR  80-1 

Ik  Dec.  1962 

(1)  Procedures  for  periodic  management  review  will 
be  developed  and  applied  to  the  management  activity 
at  individual  systems  management  level.  Review 
points  will  be  designated  as  formal  monitoring 
points  for  Air  Foroe  assessment  of  reliability  pro¬ 
grams  to  provide  a  basis  for  management  deolslons 
regarding  any  major  reorientation  of  the  relia¬ 
bility  program. 

U.h 

(2)  Major  reliability  oheck  points  or  milestones 
on  both  activities  and  results  (as  available)  will 
be  defined  and  Integrated  Into  over-all  weapon  sys¬ 
tem  program  oontrol  procedures,  including  oomputer 
or  other  meohanised  procedures. 

9.d 

*2?Dec. 

(1)  Procedures  for  perlodlo  management  review  will 
be  developed  and  applied  to  the  management  of  all 
aerospace  and  support  system  programs. 

5-a 

*.2-2 


0 


TABLE  4-4 

SUMMARY  OP  RESPONSIBILITIES 
RELATING  TO  REVIEW  OP  RELIABILITY  PROGRAMS 


f 

t 


f 


Air  Force 
Document 

Level 
to  Which 
Assigned 

Statement  of  Responsibility 

Pa,r , 

No. 

APR  80-5 

4  June  1962 

APSC 

(1)  Assume  responsibility  for  all 
reliability  design  and  development 
testing,  provide  technical  guidance 
to  the  program,  and  evaluate  and 
validate  the  results  on  all  systems. 

6.e 

(2)  Collaborate  with  the  operating 
commands  to  verify  achieved  relia¬ 
bility  in  the  operational  environ¬ 
ment  during  operational  testing  of 
systems . 

6.f 

Operating 

Commands 

(l)  Participate  with  APSC  and  AFLC 
in  the  periodic  formal  review  of 
reliability  programs. 

8.d 

AFSCR  80-1 

14  Dec.  1962 

System 

Program 

Offices 

(SPO's) 

(1)  Maintain  an  adequate  staff  for 
monitoring  of  the  reliability  pro¬ 
gram  for  the  system  with  which  they 
are  concerned. 

7.a 

CMR 

(l)  Obtain  objective  evidence  to 
determine  compliance  or  degree  of 
non-compliance  with  contractual 
reliability  requirements . 

8.b 

AFBSDR  80-5 

28  Dec.  1962 

SPO 

Reliability 

Coordinators 

(1)  Implement  a  system  mathematical 
model  for  the  perlodlo  assessment 
of  system  reliability  and  solution 
of  problem  areas. 

6.b.(3) 
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4. 2. 4.1  Periodic  Program  Reviews 


It  Is  a  common  practice  in  most  programs  for  the  Air 
Force  and  the  contractor  to  meet  periodically  for  program 
reviews.  Depending  on  the  procedures  established,  these 
reviews  may  be  concerned  solely  with  the  reliability  pro¬ 
gram  or  may  consider  that  program  as  only  one  factor  among 
many.  At  such  meetings,  the  contractor  usually  makes  an 
oral  presentation  of  program  status  and  progress,  with  the 
customer  serving  primarily  as  interrogator. 

The  procedures  associated  with  the  conduct  of  periodic 
program  reviews  are  not  stereotyped,  but  must  usually  be 
developed  for  each  individual  program.  The  factors  listed 
in  Table  4-5  are  offered  for  jjuicL-ice  in  establishing  and 
conducting  a  series  of  program  reviews. 

If  separate  technical  meetings  (see  3.1.2)  not  pri¬ 
marily  intended  for  review  purposes  are  conducted,  care 
should  be  taken  to  insure  distinction  between  the  two 
procedures.  Also,  care  must  be  taken  to  insure  that 
surveillance  activities  (see  4.1)  and  periodic  status 
reporting  activities  (see  3.1.3)  complement  rather  than 
duplicate  the  periodic  program  review  activities. 

Program  reviews  in  which  the  Air  Force  is  a  partici¬ 
pant  (as  observer,  interrogator,  or  director)  should  be 
Independent  of  and  in  addition  to  any  reviews  conducted 
internally  by  the  contractor. 


4. 2. 4. 2  Major  Milestone  Reviews 


Major  milestone  reviews  are  similar  to  perlodlo 
program  reviews  in  that  the  cumulative  requirements  of 
the  program  represent  the  area  of  Interest.  They  differ 
from  the  periodic  reviews,  however,  to  the  extent  that 
the  milestone  reviews  are  usually  more  comprehensive  and 
Involve  a  deliberate  investigative  effort  oonduoted  by 
the  SPO  Reliability  Coordinator. 

The  approach  to  performing  major  milestone  reviews 
requires  essentially  the  same  processes  as  the  implementation 
of  program  surveillanoe :  the  SPO  Reliability  Coordinator 
must  develop  a  plan,  exeoute  that  plan,  and  feedbaok  results 
into  the  program.  In  developing  the  plan  for  milestone 
reviews,  the  Reliability  Coordinator  must  determine  the 
elements  of  work  to  be  surveyed,  determine  the  number  of 
milestones  to  be  reviewed,  and  establish  standards  for  the 
reviews. 
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TABLE  4-5 

FACTORS  TO  BE  CONSIDERED 
IN  CONDUCTING  PROGRAM  REVIEWS 


(1) 


Faotor 


Requirement  or  Responsibility 


Conduot  of 
program  review 


Should  be  specified  In  the  con¬ 
tract  statement  of  work. 


(2)  Frequency  of 
reviews 


(3)  Preparation  of 
agenda 


Should  be  decided  on  basis  of 
comprehensiveness  of  the  relia¬ 
bility  program. 

Should  be  responsibility  of  con¬ 
tractor  and  should  be  provided 
to  the  Air  Force  at  some  prescribed 
time  prior  to  the  review. 


(4)  Presentation 


Should  be  responsibility  of 
contractor . 


(5)  Representation 
of  contractor 


Should  include  reliability 
management  personnel. 


(6)  Content  of  review 


Should  include  all  major  elements 
of  the  reliability  program. 


(7)  Published  synopsis 
of  review 


Should  be  furnished  to  Air  Foroe 
by  contractor  at  some  prescribed 
time  after  the  review  and  should 
list  all  aotion  items  established. 


(8)  Review  duration 


Should  depend  on  frequency,  but  if 
meetings  are  held  monthly,  it  is 
advisable  to  confine  their  duration 
to  one  or  two  days. 
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4.2,4. 2.1  Elements  of  Work  to  Be  Surveyed 


Ideally,  as  In  the  surveillance  task  (see  4. 1.4. 1.1) 
the  elements  of  work  which  are  to  be  surveyed  at  each  * 
major  milestone  In  a  reliability  program  should  have  been 
pre-established  by  the  requirements  of  the  contract  work 
statement  and/or  the  c.ontractor's  reliability  program  plan. 


4. 2.4. 2. 2  Number  of  Major  Milestones 
to  Be  Reviewed 


Existing  Air  Force  policy  (see  Table  4-3)  stipulates 
that  major  program  milestones  for  review  shall  be  pre- 
established.  Table  4-6  Illustrates  three  distinct  methods 
which  can  be  used  to  identify  major  milestones,  although 
many  other  ways  are  also  available. 

There  is  no  one  listing  of  major  program  milestones 
which  is  applicable  to  all  programs.  Hence,  it  is  advis¬ 
able  for  the  SPO  Reliability  Coordinator  to  establish 
milestone  review  requirements  for  a  specific  reliability 
program  at  his  own  discretion.  He  should  be  guided  in 
this  task  by  the  following  factors: 

(1)  Any  set  of  milestones  established  for  major 
reliability  review  should  be  consistent  with 
milestones  established  for  the  over-all 
program , 

(2)  The  milestones  should  represent  significant 
and  definable  points  in  time  with  respect  to 
the  system  life  cycle. 

(3)  The  milestones  should  be  confined  to  that 
period  of  time  during  which  the  SPO  Reliability 
Coordinator  is  actively  concerned  with  the 
program. 

(4)  The  number  of  milestones  should  be  held  to  a 
practical  minimum  so  that  sufficient  attention 
oan  be  devoted  to  the  planned  review. 

(5)  Establishment  of  major  milestone  reviews  should 
take  into  consideration  the  frequency  of  periodio 
program  reviews . 

(6)  Each  individual  major  milestone  review  should  be 
characterised  by  preliminary  planning  effort  such 

as  notifying  the  oontraetor,  Issuing  instructions 
to  him,  etc. 
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4. 2. 4. 2. 3  Standards  for  Major  Milestone 
Reviews 


The  approach  to  establishing  appropriate  standards 
for  milestone  reviews  Is  similar  to  that  discussed  under 
4. 2. 4*2.1  (standards  for  surveillance)  except  that,  in 
this  case,  the  standards  must  additionally  be  related  to 
the  pre-established  milestones.  To  illustrate,  referenoe 
is  made  to  the  matrix  in  Figure  4-2 .  Here  it  is  assumed 
that  the  reliability  program  under  consideration  has  been 
arbitrarily  divided  into  thirteen  elements  of  work  and 
that  the  system  life  cycle  is  described  by  the  conceptual, 
acquisition,  and  operational  phases.  Each  block  of  the 
matrix,  then,  defines  the  basis  for  a  set  of  standards 
which  must  be  developed. 

In  4. 1.4. 1.2,  it  was  suggested  that  the  SFO  Relia¬ 
bility  Coordinator  compile  a  Standards  Notebook  for 
guiding  his  surveillance  activities.  The  same  format 
could  be  utilized  in  compiling  standards  for  each  major 
milestone  review. 


4.2.5  Additional  References 
on  Program  Review 


Reliability  Management  by  Objectives  and  Results, 

Ii.  Ball,  Eighth  National  Symposium  on  Relia- 
blllty  and  Quality  Control,  January  1962,  pp.  156-162. 

Management  Policies  for  Assigning  Departmental 
KeiraCTTt'y'Re spon's ibil'l tie's?  ZrV 7  BmrinduB trial 
^uaIT€y”C6nl:roI7 ' Vol .  T T7 No.  ■  10  (April  1961), 
pp.  16-19. 

Reliability  Audits.  R.  R.  Landers,  Maohlne  Design, 
75TT337  No.  5‘‘(March  2,  1961),  pp.  76-83. 

Methods  for  Evaluating  Reliability  Qrowth  and 
Ultimate  Reliability  During  Development  ofT" 

Complex  System,  I.  ft.  Saits,  Fifth  national 
Symposium on  Reliability  and  Quality  Control, 

January  1959,  PP.  89-97. 

Keeping  Soore  on  Reliability.  E.  F.  Dertinger  and 
R.  P.  Bosely,  flhird  National  Symposium  on  Reliability 
and  Quality  Control,  January  1957*  pp*  49-52 . 
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FIGURE  4-2 

MATRIX  RELATING  ELEMENTS  OF  WORK 
IN  RELIABILITY  PROGRAM 

WITH  PRE-ESTABLISHED  PHASES  OF  SYSTEM  LIFE  CYCLE 


Conceptual 

Phase 

Definition 

Phase 

Acqulstlon 

Phase 

Operational 

Phase 

General  Program  Considerations 

Numerical  Reliability  Requirements 

Program  Reviews 

Reliability  Interface  with 

Total  Program 

Design  Reviews 

Design  Change  and  Control 

Mathematical  Models 

Test  Planning 

Exchange  of  Data 

Problem  Control 
and  Corrective  Action 

Reliability  Demonstration 

Failure  Reporting  System 

Reports  Submittal 
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5-  EXPLANATION  OF  TERMS  AND  ABBREVIATIONS 
USED  IN  HANDBOOK 


A  number  of  terms  and  abbreviations  used  throughout 
this  handbook  have  specialized  meanings  which  may  not  be 
familiar  to  all  users  of  the  handbook.  This  appendix 
presents  the  definitions  and  identifications  which  are 
intended  in  the  discussions  of  the  present  text.  It  is 
recognized,  of  course,  that  in  rapidly  expanding  fields 
such  as  reliability  engineering,  aerospace  technology, 
etc., the  existence  of  different  definitions  for  the  same 
technical  term  is  not  an  uncommon  occurrence. 

(See  MIL- STD  721A,  2  August  1^62,  Definition  of  Terms  for  Reliability 
Engineering) 
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EXPLANATION  OF  TERMS 


Acquisition 

phase 


Adjustment 
( application 
factor) 

Analysis 

of  variance 


Apportionment 


Availability 


Catastrophic 

failure 


Complexity 

level 

(eq;uipnent) 


The  period  starting  after  the  issuance  of 
the  SP  Directive  until  the  acceptance  by 
the  user  of  the  la6t  operating  unit  in  a 
certain  series,  or  until  the  Specific 
Operational  Requirement  has  been  demon¬ 
strated  through  Category  II  testing  and 
all  required  updating  changes  resulting 
from  the  testing  have  been  identified, 
approved,  and  placed  on  procurement,  which¬ 
ever  occurs  later. 

A  multiplier  applied  to  data  to  adjust 
for  variable  conditions. 


The  separation  of  sum-of- square  variations 
from  the  mean  into  components  which  can  be 
assigned  to  variations  between  classes,  or 
subclasses,  of  appropriate  data.  Constit¬ 
uent  portions  of  sums  of  squares  indicate, 
through  mean  squares,  the  magnitude  of  class 
differences.  The  extent  to  which  they  vary 
from  the  residual  mean  square  is  a  test  of 
the  hypothesis  that  such  differences  are 
governing  the  situation. 

Allocation  or  assignment  of  the  system  relia¬ 
bility  requirements  to  subsystems  and  com¬ 
ponents  in  accordance  with  their  complexity 
and  other  factors,  in  such  fashion  that  if 
each  component  or  subsystem  meets  its  allocated 
requirement,  the  system  reliability  require 
ment  will  be  met. 

Availability  is  the  probability  that  the 
system  is  operating  satisfactorily  at  any 
point  in  time  when  used  under  stated  condir 
tions  where  the  total  time  considered  Includes 
operating  time,  active  repair  time,  adminis¬ 
trative  time,  and  logistic  time.  (See 
Appendix  F  for  definition  of  Intrinsic  Avail¬ 
ability  and  Operational  Readiness) 

A  sudden  change  in  the  operating  characteristic 
of  an  item  resulting  in  a  complete  lack  of 
useful  performance  of  the  item. 

The  figure  of  merit  or  measure  of  the  quantity 
of  related  parts  of  circuits.  The  total  number 
of  electronic  parts  (n)  is  often  used  as  a 
measure.  Complexity  units  are  sometimes  used 
as  a  preliminary  and  approximate  measure. 
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Explanation  of  Terms  (Continued) 


Failure  mode 


Failure  rate 


Feasibility 

study 


Flight 

proofing 

tests 


Hazard  rate 


Human 

faotora 


The  physical  description  of  the  manner  in 
which  a  failure  occurs  and  the  operating 
condition  of  the  equipment  or  part  at  the 
time  of  the  failure . 

The  number  of  failures  per  unit  time  of  a 
specified  item. 

The  study  of  a  proposed  item  or  technique 
to  determine  the  degree  to  which  it  is 
practicable,  advisable,  and  adaptable  for 
the  intended  purpose . 

Tests  based  on  environmental  stress-strain 
safety  margins,  with  the  equipment  subjected 
to  increasing  levels  of  environmental  stress. 

If  no  failures  occur  under  the  conditions 
imposed,  the  design  is  considered  adequate. 

All  equipment  used  is  considered  expended  and 
may  not  be  used  in  the  field. 

These  tests  and  the  qualification  tests 
usually  fulfill  the  environmental  qualification 
requirements  for  R  and  D  equipment.  Safety 
margins  used  are  above  the  expected  in-service 
levels  to  allow  for  the  variability  of  in- 
service  conditions  and  the  individual  character¬ 
istics  of  the  equipment.  The  test  conditions 
should  not  be  so  severe  as  to  exceed  reasonable 
safety  margins  or  to  excite  unrealistic  or 
improbable  modes  of  failure. 

The  instantaneous  failure  rate.  It  is  defined 
as  the  limit  of  the  failure  rate  as  the 
interval  length  approaches  zero. 

z(t)  ■  u( t)  where  R(t)  «  u(t)dt 

KTw  A 

A  constant  hazard  rate  is  equivalent  to  the 
exponential  case . 

Facts  about  human  behavior  which  affeot  the 
design  of  systems.  As  a  discipline,  its  goal 
is  to  achieve  an  optimal  system  with  an 
efficient  man  working  in  a  safe  and  habitable 
environment. 
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Explanation  of  Terms  (Continued) 


Incentive  A  System  contract  wherein  the  fee  or  profit 

contract  margin  is  a  function  of  the  demonstrated 

(fteliability)  reliability  of  the  system  produced  by  the 
contractor.  The  maximum  fee  is  only 
attainable  by  the  demonstrated  reliability 
exceeding  a  stated  contractual  reliability 
specification.  i 


Infant  Failures  occurring  during  the  debugging 

mortality  phase  or  early  portion  of  equipment  life, 

failures 

(early  life) 

Maintainability  Maintainability  is  the  probability  that, 
when  maintenance  action  is  initiated  under 
stated  conditions,  a  failed  system  will  be 
restored  to  operable  condition  within  a 
specified  total  down  time. 


Mean  life  The  arithmetic  average  of  the  lifetimes  of 

all  items  considered.  The  mean  life  values 
have  meaning  only  in  relation  to  the  type 
of  frequency  distribution  assumed  by  the  data. 


Mean-time- 
be tween- 
failure 


Mean-time- 

to-repair 

Milestone 


Mission 


The  sum  of  individual  operating  times  accumulated 
by  a  population  of  identical  equipment  items, 
divided  by  the  number  of  failures  occurring 
in  the  time  of  observation.  The  operating 
time  must  include  that  accumulated  by  items 
which  did  not  fall,  as  well  as  the  time 
accumulated  on  items  which  did  fail. 

The  average  amount  of  time  spent  in  correcting 
malfunctions. 

A  significant  and  frequently  critical  event 
having  qualitative  and/or  quantitative  aspects 
which,  when  completed,  indicates  the  accomp¬ 
lishment  of  a  step  in  progress  toward  a  pre¬ 
determined  goal. 

A  coordinated  execution  of  functions  according 
to  a  preconceived  tactical  operations  plan  for 
the  accomplishment  of  an  objective  against  the 
enemy. 
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Explanation  of  Terms  (Continued) 


Operational 

phase 


Parametrlo 

study 


The  period  from  aooeptanoe  by  the  user  of 
the  first  operating  unit  until  disposition 
of  the  system.  The  operational  phase  over¬ 
laps  the  acquisition  phase. 

A  determination  of  the  relationships  exist¬ 
ing  between  the  parameters  of  a  system. 


Prediction  The  estimation  of  the  future  behavior  of  a 
system  on  the  basis  of  knowledge  of  Its 
parts,  functions,  operating  environments, 
and  their  interrelationships. 


Probability 

A  measure  of  the  likelihood  of  occurrence 
of  a  particular  event.  The  ratio  of  the 
number  of  ways  a  particular  event  can  ooour 
to  the  total  number  of  events  whloh  can 
possibly  occur. 

Probability 
of  failure 

The  likelihood  that  an  item  will  fail  during 
a  specified  period  of  time  in  a  given 
environment . 

If  Pf  «  probability  of  failure  and 

P_  -  probability  of  success,  then: 

Pf  -  1  -  P.  or  Pf  +  P,  -  1 

Probability 
of  suooess 

The  likelihood  of  successful  operation  of  an 
Item  for  a  specified  period  of  time  under  a 
specified  environment-.  The  complement  of 
the  probability  of  failure. 

Probability 
of  survival 

A  numerical  expression  of  reliability.  It  has 
the  aooepted  nomenclature  of  P8  and  a  range  of 

0  to  1.0  where  0  and  1  indicate  the  extremes 
of  "Impossibility"  and  "oertalnty," respectively 
In  other  words,  the  probability  of  a  given 
equipment  performing  its  intended  function  or 
the  given  use  cycle. 

Quality 

oontrol 

A  factory- oriented  operation  to  assure  manu¬ 
facture  of  a  uniform  produot  within  specified 
defeot  limits  in  aooordanoe  with  design 
requirements . 

3 
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Explanation  of  Terms  (Continued) 


Redundancy  The  existence  of  more  than  one  means  for 
accomplishing  a  given  task,  where  all 
means  must  fail  before  there  is  an  over¬ 
all  failure  to  the  system. 

Reliability  The  probability  that  an  item  will  perform 
a  required  function  under  specified  con¬ 
ditions  for  a  specified  period  of  time. 

Reliability  A  periodic  record  of  reliability  estimates 
growth  curve  by  which  trends  can  be  observed  and 
reliability  growth  determined. 

Reliability  Figures  of  merit,  such  as  ratios,  factors, 
index  etc.,  used  to  denote  relative  reliability. 

Service  Service  readiness  is  the  probability  that 

readiness  at  any  point  in  time,  the  system  is  either 

operating  satisfactorily  or  ready  to  be 
placed  in  operation  on  demand  when  used 
under  stated  conditions,  including  stated 
allowable  warning  time.  The  total  calendar 
time  is  the  basis  for  computation  Of  service 
readiness . 

A  technique  somewhat  similar  to  structural 
stress  analysis,  applied  to  equipment  design 
to  evaluate  the  influence  of  environmental, 
functional,  and  time  stresses,  as  an  aid  to 
assuring  design  integrity  or  predicting 
reliability. 

Subcontractor  A  supplier  of  subsystems  or  equipments  to 
the  contractor. 

System  System  effectiveness  is  the  probability  that 

effectiveness  a  system  (or  a  system  complex)  will  accomplish 
a  stated  mission.  (If  the  duration  of  time  is 
important,  it  must  be  included  in  mission  definition) 

System  The  complete  life  cycle  of  a  system  including  conceptual 

life  cycle  phase,  acquisition  phase,  and  operational  phase. 


Stress 

analysis 
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Trade-off 


Vendor 


Explanation  of  Terma  (Continued) 


The  prooedure  of  trading  a  degree  of  one 
attribute  to  gain  a  degree  of  another 
attribute.  E.g.,  a  degree  of  reliability 
might  be  sacrificed  to  obtain  a  greater 
degree  of  performance  under  oertaln  condi¬ 
tions,  or  vloe  versa. 

A  supplier  of  parts  or  other  Items  of  low 
complexity  levels  to  the  oontraotor. 
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ADO 

AFBSD 

AFLC 

AFPI 

ASTRO 

AFSC 

AFSSD 

AGE 

AGREE 

ASPR 

ASTIA 

BSD 

CMR 

DOD 

DP 

GFE 

IDEP 

IRE 

RASA 

OPR 

PEP 


EXPIANATION  OF  ABBREVIATIONS 

Advanced  Development  Objective 

Air  Force  Ballistic  System  Division 

Air  Force  Logistic  Command 

Air  Force  Procurement  Instruction 

Air  Force  Plant  Representative  Office 

Air  Force  System  Command 

Air  Force  Space  System  Division 

Aerospace  Ground  Equipment 

Advisory  Group  on  Reliability  of  Electronic 
Equipment,  Office  of  Assistant  Secretary 
of  Defense  (Research  and  Engineering) 

Armed  Services  Procurement  Regulation 

Armed  Services  Technical  Information  Agency 

Ballistic  System  Division 

Contracts  Management  Region 

Department  of  Defense 

Development  Plan 

Government  Furnished  Equipment 

Interservice  Data  Exchange  Program 

Institute  of  Radio  Engineers 

National  Aeronautics  and  Space  Administration 

Office  of  Primary  Responsibility 

Product  Evaluation  Program 
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PERT 

Program  Evaluation  and  Review  Technique 

PSPP 

Proposed  System  Package  Plan 

t  RPP 

Request  for  Proposal 

RMI 

Reliability  Maturity  Index 

SE/TD 

Systems  Englneerlng/Teohnloal  Dlreotion 

SOR 

| 

Specific  Operational  Requirement 

I  .  SPO 

System  Project  Offloe 

SPP 

System  Package  Plan 

SSD 

Space  System  Division 

6.  INDEX  DESCRIPTION  AND 
INSTRUCTIONS  FOR  USE 


Thia  index  ia  applicable  to  both  Volumes,  1  and 
2,  of  the  Reliability  Management  Handbook. 
Index  entries  with  number-prefix  page  numbers 
apply  to  Volume  I;  the  entries  with  letter-prefix 
page  numbers  apply  to  Volume  II. 


INDEX 


Acceptance  tests,  see  testing 
Accelerated  tests,  see  testing 
Achieved  reliability,  measurement  of 
Air  Force  policy,  A. 4 -29 
definition,  A. 4-29 

relationship  between  achieved  and  predicted 
reliabilities,  A. 4 -29 
use  of  failure  data,  A. 4-29 
Aerospace  Corporation,  services  of 
contractor  surveillance,  3 • 3-8 
environmental  studies,  3*3-8 
failure  analysis,  3*3-8 
reliability  data  retrieval,  3.3-9 
statistical  analysis,  3*3-7 
system  analysis,  3*3-8 

AFPRO  (Air  Force  Plant  Representative  Office), 

3.1-32,  3*1-34,  4.1-8 
Air  Force  policy 

acceptance  tests,  A.4-16 

Air  Force/contractor  liaison,  3*1-29 

application  of  reliability  technology,  research  on,  A. 1-20 

bidder  selection,  2.2-3 

contract  negotiation,  2.2-25 

contractor  data  submittals,  3.1-23 

design  reviews,  A. 4-7 

developmental  tests,  A. 3-2 6 

documentation  for  product  development,  A. 3-47 

docvment  review  in  contracted  program,  A. 4-2 

environmental  research,  A. 1-10 

environmental  studies,  A. 3 -42 

failure  analysis,  A. 3-28 

failure  corrective  action,  A. 3-28 

failure  rate  data  research,  A. 1-11 

failure  reporting,  A. 3-28 

formal  reliability  training,  A. 2-3 

inspection  for  compliance  to  documentation,  A. 4-25 

materials  research,  A. 1-6 

measurement  of  achieved  reliability,  A -4-29 

motivational  reliability  training,  A. 2-13 

parts  reliability  data,  A.  1-11,  A. 3-15 

physics  of  failure  research,  A.l-16 

product  quality  control,  A. 4 -25 

program  controls 

for  Air  Force  use,  3*2-1 
for  contractor  use,  A. 5-20 
program  funding,  3.4-1 
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Air  Force  policy  (continued) 
program  plan,  2.3-2,  A.  5-4 
program  review 

by  Air  Force,  4.2-2 
by  contractor,  A.5-15>  A. 5-16 
program  surveillance  (monitoring) 
by  Air  Force,  4.1-1 
by  contractor,  A.5-15 
program  technical  direction,  3*3-1 
proposal  evaluation,  2.2-16 
reliability  apportionment,  A. 3-10 
reliability  demonstration,  A.4-18 
reliability  prediction,  A. 3-2 
reliability  requirements 

qualitative  requirements,  2.1-32,  2.1-35 
quantitative  requirements,  2.1-4 
statement  of  requirements,  2.1-48 
reliability  status  reporting 

contractor  reports,  3.I-3,  A. 5-10 
internal  Air  Force  reports,  3*1-35 
Request  for  Proposal,  2.2*9 
statistical  methods  research,  A. 1-2 
vendor  selection  and  control,  A. 3-53 
Air  Force  reliability  regulations,  1-5,  B-l 
Allocation  of  reliability,  see  apportionment 
Application  of  reliability  technology,  research  on 
Air  Force  policy,  A. 1-20 
definition,  A. 1-20 
major  tasks,  A. 1-21 
Apportionment  of  reliability 
Air  Force  policy,  A.3-IO 
data  requirements,  A. 3 -11,  A.3-I3 
definition,  A. 3 -10 

number  and  frequency  of  apportionments,  A. 3-11,  A. 3-13 
Approved  parts  list,  A.3-I8 
Assurance  of  reliability,  A. 4-1 
Assurance  tests,  see  testing,  acceptance  and 
qualification  tests 

ASTIA  (Armed  Services  Technical  Information  Agency),  3*4-9 
Audit  of  reliability  program,  see  program  review 
Availability/reliability  relationship,  2.1-14,  2.1-16 

■Bidders'  conference,  2.2-12 
Bidder  selection 

Air  Force  policy,  2.2-3 
evaluation  procedures,  2.2-5,  2.2-8 
reliability  evaluation  questionnaire,  2.2-7 
responsibilities  for,  2.2-2>  2.2-4 
sources  of  bidder  information,  2.2-5 
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Communications ,  Air  Force/contractor,  see  reports;  technical 
meetings,  data  submittals,  status  reporting^ liaison 
Complexity  (equipment) /reliability  relationship, 

2.1-12,  2.1-25,  A.4-23 

Component  reliability  status  document,  3 -2-8 
Confidence  levels,  see  statistical  confidence  levels 
Contract  negotiation,  see  contracting,  Air  Force/prim e 
contractor 

Contract  work  statement,  see  statement  of  reliability 
requirements 

Contracting,  Air  Force/prime  contractor 
Air  Force  policy,  2.2-25 
background  tasks,  2.2-24 
contractor  commitments,  A. 5-3 
definition,  2.2-24,  A-5-2 

negotiation  actions  and  meetings,  2.2-27,  2.2-29 
non-negotiable  areas,  2.2-28 
responsibilities  for,  2.2-26 
Contracting,  prime  contractor/vendor,  A. 3-21 
A.3-55,  A. 3-56,  A. 5-2 
Contractor  selection,  2.2-1 

also  see  bidder  selection,  proposal  evaluation, 
contracting 

Control  charts,  in  program  management,  3.2-4,  3.2-9,  3. 2-10,  A. 5 -22 
Cost  (system) /reliability  relationship  2.1-17,  2.1-18,  3.4-4,  3-4-5 

Data  interchange 

failure  rate  and  other  reliability  data,  3-3-9,  3-4-8,  3-4-9, 

A. 1-12,  A. 1-13,  A. 3-20,  A. 3-21 
materials  research  data,  A. 1-8 
Data  submittals,  contractor's 

Air  Force  administrative  responsibilities,  3.1-24,  3-1-27 
Air  Force  policy,  3-1-23 
definition,  3-1-22 
requirements  for,  3-1-25 
review  by  Air  Force,  3-1-25 
Demonstration  of  reliability 
Air  Force  policy,  A.4-18 
assumed  failure  distribution  in,  A. 4-23 
curtailment  of  testing,  A. 4-24 
definition,  A.4-18 
effect  of  complexity  level.  A. 4-23 
failure  definition  for,  A. 4-23 

planning  and  scheduling,  A. 4-19,  A. 4-21;  A. 4-22,  A. 4-23 
relationship  between  test  parameters  and  reliability,  2.1-23 
stating  requirements  for,.  2.1-20,  2.1-54 


6-4 


t 


Design  review 

Air  Force  policy,  A. 4-7 
content  of  reviews,  A. 4-11,  A. 4-12 
customer  attendance  at  reviews,  A.4-14 
definition,  A. 4 -7 
documentation  of  reviews,  A. 4-11 
funding  of  reviews,  A. 4-13 
responsibilities  for  reviews,  A. 4-8,  A. 4-9 
scheduling  of  reviews,  A. 4-10 
trade-offs  resulting  from  reviews,  A.4-14 
use  of  check  lists  in  reviews,  A. 4-13 
vendor  and  subcontractor  reviews,  A.4-14 
Developmental  tests,  see  testing 

Development  time/reliability  relationship,  2.1-12,  2.1-13 
Documentation 

Air  Force  reliability  regulations,  1-5,  B-l 
environmental  criteria  document,  A. 3-42 
for  design  reviews,  A. 4-11 
for  development  of  contracted  product 
Air  Force  policy,  A. 3-47 
control  of  document  quality,  A. 3-51 
definition,  A. 3-47 

need  for  documentation,  A. 3 -48,  A. 3 -51 
responsibilities  for  documents,  A. 3-49 
scheduling  of  documents,  A. 3-50 
for  program  administration,  A. 5-24,  A. 5-26 
inspection  for  compliance  to  (product  quality 
control),  A. 4-25 

review  for  reliability  considerations,  A. 4-2 
also  see  status  reporting;  data  submittals;  reports 

Environment/reliability  relationship,  2.1-19,  2.1-21 
Environmental  studies 

Aerospace  Corporation  services,  3*3-8 
environmental  criteria  document,  A. 3-42 
Air  Force  policy,  A. 3-42 
definition,  A. 3-42 

designing  to  meet  environment,  A. 3-45 
determination  of  environment,  A. 3 -44 
environmental  controls,  A. 3-45 

laboratory  simulation  of  environmental  conditions,  A. 3-44 
Environments!,  tests,  see  testing 

Evaluation  of  reliability  program,  see  program  review 
Evaluation  tests,  see  testing,  developmental  tests 
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Failure  analysis 

Aerospace  Corporation  services,  3*3*8 
Air  Force  policy,  A. 3 -28 
analysis  procedures,  A. 3-35 
control  of  analysis  activity,  A. 3-40 
effect  of  failure  on  mission  reliability,  A. 3-37 
items  to  be  analyzed,  A. 3-36 
recurrent  failures,  A. 3 -37 
Failure  corrective  action 
Air  Force  policy,  A. 3 -28 
implementation,  A. 3-38 
Failure  definition,  A. 3-4,  A. 3-32,  A. 4-23 
Failure  distribution 

assumed  for  reliability  demonstration,  A. 4-23 
failure  frequency/type  relationship,  A. 3-28,  A. 3-29 
Failure  mode,  determination  of,  A.l-13,  A.l-17,  A. 3-26,  A. 3-36 
Failure  physics,  see  physics  of  failure  research 
Failure  rate  data 

Air  Force  policy  relative  to  use,  A. 1-11,  A.l-16,  A.3-I5 
in  reliability  predictions,  A.3-3>  A. 3-5,  A. 3-8 
incomplete  data,  use  of,  A.l-14,  A. 3-8 
interchange,  3*3-9,  3*4-8,  A.l-13,  A. 3-20,  A. 3-21 
laboratory  data,  use  of,  A. 3-8,  A.3-I3,  A. 3-23 
Failure  rate  data  research 
Air  Force  policy,  A. 1-11 
definition,  A. 1-11 
major  tasks,  A. 1-12,  A.l-16,  A. 3-17 
Failure  reporting 

Air  Force  policy,  A. 3-28 
collection  of  data,  A. 3-33 

compatibility  of  customer/contractor  systems,  A. 6-7 
control  of  reporting  activity,  A. 3-39 
items  to  be  reported,  A. 3-31 
reporting  forms,  A.3-3I,  A. 3 -39 
reporting  system  and  procedures,  A. 3-32,  A. 3-34 
use  of  data  in  measuring  achieved  reliability,  A. 4-30 
Field  tests,  see  testing 
Films,  reliability  engineering,  3*4-7,  A. 2-13 
Formal  reliability  training,  see  training 
Funding  of  reliability  programs,  see  program  support 

Human  factors,  effect  on  reliability,  A. 4-26 

H3EP  (interservice  Data  Exchange  Program),  3. 3-9,  3*4-8,  A. 1-12, 
A.l-13,  A. 3-20,  A. 3-21 
Information  services,  3*4-8,  3*4-9 
Installation  manual,  A. 6-3 


Liaison,  Air  Force/contractor 
Air  Force  policy,  3 •1-29 
definition,  3*1-29 
responsibilities  for,  3*1-31 
SPO/AFPRO  coordination,  3-1-32,  3*1-34,  4.1-8 
visits  to  contractor  facilities,  3*1-30,  3*1-32 

Maintainability/ reliability  relationship,  2-1-14,  2-1-16 

Maintenance  manual,  A. 6-5 

Management  controls,  see  program  controls 

Manuals 

for  equipment  use  and  maintenance,  A. 6-3 
for  reliability  training,  A. 2-8,  A. 2-10 
Manufacturing  control,  A. 4-25 
Materials  classification,  A. 1-7 
Materials  research 

Air  Force  policy,  A. 1-6 
definition,  A. 1-6 
factors  to  be  studied,  A. 1-7 
Mathematical  reliability  models,  A. 3-5 
Measurement  of  reliability,  see  achieved  reliability 
Meetings,  Air  Force/contraetor 
bidders'  conference,  2.2-12 
contract  negotiation  meetings,  2.2-27,  2-2-29 
technical  meetings  for  specific  program,  3*1-18 
also  see  liaison 

Milestone  charts,  for  program  control  and  review, 

3*2-9,  3*2-11,  4.2-4 

MEL-R-27542  cross  reference  chart,  1-8,  1-9,  A-l,  A-2 
Mission  profile,  description  of,  2.1-7,  A. 3-4 
Monitoring  of  reliability  program,  see  program  surveillance 
Motivational  reliability  training,  see  training 

Negotiation  of  contracts,  see  contracting 
Networks,  PERT,  3*2-5 

Numerical  reliability  requirements,  see  quantitative 
requirements 

Operator's  manual,  A. 6-1 

Organization  charts,  in  program  management,  A. 5-22 

Parts  improvement,  A. 3-18 
Parts  program 

Air  Force  policy,  A.3-I5 
definition,  A.3-I5 
major  tasks,  A. 3-16 
vendor  parts  program,  A. 3-21 
Parts  reliability  data,  see  failure  rate  data 
Parts  selection  and  application,  A. 3-1?,  A. 3-18,  A. 3-24 
Parts  specifications,  A. 3-21 
Parts  testing  control,  A. 3-19 
Parts  tests,  see  testing 


6-7 


? 


S' 


f 


I 


PEHT  (Program  Evaluation  and  Review  Technique) 
advantages,  3.2-7>  A. 5-22 
associated  terms,  3.2-4 
method  of  use,  3*2-6 
networks,  3*2-5 
Physics  of  failure  research 
Air  Force  policy,  A.l-16 
definition,  A.l-16 
major  tasks,  A. 1-17 
planning,  A.l-18 
reporting,  A.l-18 
Prediction  of  reliability 
Air  Force  policy,  A. 3-2 
data  requirements,  A. 3-4,  A. 3-7 
definition,  A. 3 -2 

number  and  frequency,  A. 3-3,  A. 3-7 
procedures ,  2.1-24,  A. 3-8 
reporting  requirements,  A. 3-5 
use  of  prediction  results,  A. 3-6,  A. 3-9 
Preferred  parts  list,  A. 3-18 
Production  control,  A. 4-25 
Program  administration 
by  Air  Force,  3-1 
by  contractor,  A. 4-4 

also  see  program  controls,  plan, review,  surveillance; 
status  reporting;  documentation;  contracting 
Program  controls 

for  Air  Force  use 

Air  Force  policy,  3*2-1 

definition,  3.2-1  _ 

formal  control  systems  (PERT,  Ml),  3*2-4,  3*2-7 
responsibilities  for,  3.2-2,  3-2-13 
simplified  control  devices,  3*2-8 
for  contractor  use 

Air  Force  policy,  A. 5-20 
definition,  A. 5-20 

types  of  controls,  A. 5-21,  A. 5-22,  A. 5-23 
Program  documentation,  see  documentation  for  program 
administration 

Program  expenditures  plot,  for  program  control,  3*2-10,  3*2-12 
Program  milestones 

as  basis  for  program  review,  4.2-4 
identification  of,  4.2-7 
milestone  charts  for  program  control,  3*2-9 
use  in  stating  reliability  requirements,  2.1-39 
Program  monitoring,  see  program  surveillance 
Program  office,  contractor's,  responsibilities  of,  A. 5-1,  A.5-5 
A.5-11 
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Program  plan 

Air  Force  instructions  to  contractor,  2-3-6,  2-3-9 
Air  Force  policy,  2.3-2,  A. 5-4 
content  requirements,  2.3-9,  2.3-15,  A. 5-6 
definition,  A. 5-4 

evaluation  by  Air  Force,  2.3-H,  2-3-15 
official  status,  2-3-1,  A. 5-8 
preparation  by  contractor,  A. 5-5 
responsibilities  for  enactment,  2-3-3,  2-3-4 
Program  review 
by  Air  Force 

Air  Force  policy,  4.2-2 
definition,  4.2-1 

factors  in  conduct  of  reviews.  4.2-5 
major  milestone  reviews,  4.2-4 
periodic  reviews,  4.2-4 
program  elements  to  be  reviewed,  4.2-6 
responsibilities  for,  4.2-3 
standards  for  review,  4.2-8 
by  contractor 

Air  Force  policy,  A. 5-15,  A. 5-16 
implementation,  A. 5-16,  A. 5-18 
Program  stage 

effect  on  failure  frequency/ type  relationship,  A. 3 -28,  A. 3-29 
relation  to  reliability  requirements,  2.1-29,  2.1-45,  2-3-8 
relation  to  status  reporting  requirements,  3.1-16 
Program  support  by  Air  Force 
Air  Force  policy,  3*4-1 
definition,  3-4-1 
funding 

cost  quotation  in  program  plan,  A. 5-6 
relationship  to  reliability  requirements  and 

achievement,  2.1-17,  2.1-36,  3-4-2,  3-4-4,  3-4-5,  3-4-9 
provision  of  facilities  and  equipment,  3-4-6 
provision  of  guidance  information,  3-4-6 
responsibilities  for,  3-4-2,  3-4-3,  3-4-6,  3-4-9 
Program  surveillance 

Aerospace  Corporation  services,  3- 3-8 
by  Air  Force 

Air  Force  policy,  4.1-1 
definition,  4.1-1 

program  elements  to  be  surveyed,  4.1-3 
responsibilities  for,  4.1-2 
scheduling  of  surveillance,  4.1-8 
SPO/AFPRO  coordination,  3-1-32,  3-1-34,  4.1-8 
standards  for  surveillance,  4.1-4 
by  contractor 

Air  Force  policy,  A.5-I5 
Implementation,  A.  5-16,  A.  5-18 
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Program  technical  direction 
Air  Force  policy,  3*3-1 
definition,  3*3-1 
responsibilities  for,  3*3-2 
situations  requiring  direction,  3-3-3 
use  of  outside  assistance,  3*3-6 
use  of  trade-off  curves,  3*3-5 
Progress  reports,  contractor's,  see  status  reporting 
Proposal  evaluation 

Air  Force  policy,  2.2-16 
cost  considerations,  2.2-22 
definition,  2.2-15 

evaluation  of  bidders'  reliability  capabilities,  2.2-18 
reliability  evaluation  questionnaire,  2.2-19 
responsibilities  for,  2.2-17 
Proposals,  contractor's,  see  Request  for  Proposal, 
proposal  evaluation 

Qualification  tests,  see  testing 

Qualified  bidders,  selection  of,  see  bidder  selection 
Qualitative  reliability  requirements 
Air  Force  policy,  2.1-32,  2.1-35 

applicability  of  previous  requirement  documents,  2.1-37 
definition,  2.1-31 

methods  of  stating  requirements,  2.1-38,  2.1-41,  2.1-44 
procedures  for  selection,  2.1-34 
responsibilities  for  selection,  2.1-31 
Quality  control  of  product,  see  testing,  qualification 
tests;  documentation,  Inspection  for  compliance  to 
Quantitative  reliability  requirements 
Air  Force  policy,  2.1-4 
apportionment,  A. 3-10,  A. 3-11 
confidence  levels  for,  2.1-22 
difficulty  of  achievement,  2.1-36 
in  incentive  clauses,  2.1-29 
methods  of  stating  requirements,  2.1-9,  2.1-27 
pi-ediction  for  new  systems,  2.1-24 
procedures  for  selection.  2.1-6 
reliability  growth  curve,  2.1-11 
reliability  trade-offs,  2.1-10 
responsibilities  for  selection,  2.1-5 

Recurrent  failures,  analysis  of,  A. 3-37 
Redundancy/reliability  relationship.  2.1-20 
Regulations,  reliability  (Air  Force),  1-5,  B-l 
Reliability  block  diagram,  A. 3-4 
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Reliability  Coordinator  (Air  Force  SPO),  responsibilities  of 
Air  Force/contractor  liaison,  3*1-34 
Air  Force/contractor  technical  meetings,  3*1-16 
contract  negotiation,  2.2-26 
contractor  data  submittals,  3*1-24 
coordination  Vith  ASTRO,  3*1-32,  3*1-34,  4.1-8 
enactment  of  reliability  program  plan,  2.3-il>  2.3-4 
preparation  of  reliability  requirements  statement,  2.1-49 
preparation  of  Request  for  Proposal,  2.2-9 
program  controls,  3-2-2 
program  review,  4.2-3 

program  support  activities,  3*4-2,  3*4-3,  3*4-6 
program  surveillance,  4.1-2 
program  technical  direction,  3*3-2 
proposal  reliability  evaluation,  2.2-17 
reliability  status  reporting 

administration  of  contractor  reporting  activity,  3*1-4 
preparation  of  Air  Force  internal  reports,  3*1-36 
selection  of  bidders,  2.2-2,  2.2-3 
selection  of  reliability  requirements,  2.1-5,  2.1-33 
systems  management,  1-2 

Reliability  data  interchange,  3*3-9,  3*4-8,  3*4-9,  A. 1-12,  A. 1-13 
A. 3-20,  A. 3-21 

Reliability  education  committee,  A. 2-5 
Reliability  growth  curve,  2.1-11,  3*2-10 
Reliability  measurement,  see  achieved  reliability 
Reliability  organization,  contractor's 
major  responsibilities  of,  A-4 
need  for,  A. 5-8 

Reliability  program  establishment  and  administration, 
see  various  headings  under  "program" 

Reliability  requirements 
formulation  of 

Air  Force  directives,  2.1-2 
methods,  2.1-1,  2.1-10 
inclusion  in  Request  for  Proposal,  2.2-12 
statement  in  contract,  2.1-51 

also  see  qualitative  and  quantitative  requirements, 
statement  of  requirements,  vendor  reliability 
requirements 

Reliability  survey  questionnaires,  sample  supplier,  Appendix  1) 
Reliability  trade-offs,  2.1-10 
Reports,  failure,  see  failure  reporting 
ReportB,  reliability,  3.1-26,  A. 3-5 

also  see  status  reporting,  documentation,  data 
submittals 

Reports,  test,  see  documentation,  for  tests 


Request  for  Proposal  (KFP) 

Air  Force  policy,  2.2-9 
content,  2.2-11 
definition,  2.2-9 
recipients  of,  2.2-11,  2.2-13 
responsibilities  for  preparation,  2.2-10 
Research,  reliability 

significant  contractor  activities,  A. 1-1 
also  see  headings  for  specific  types  of  research 
(e.g. ,  application  of  reliability  technology, 
environmental,  failure  rate  data,  materials, 
physics  of  failure,  statistical  methods) 

Reviews 

design,  A. 4-7 
documents,  A. 4-2 

reliability  program,  4.2-1,  A. 5-15 
Risk,  1-1,  1-13 

HMI  (Reliability  Maturity  Index),  for  program  control,  3*2-7 

Size  (system) /reliability  relationship,  2.1-17,  2.1-20 
Specifications 

Air  Force  reliability  documents,  1-5,  3*4-7,  B-l 
for  parts,  A. 3-21 

review  for  reliability  considerations,  A. 4-3 
use  in  stating  reliability  requirements,  2.1-40 
Statement  of  reliability  requirements,  contractual 
Air  Force  policy,  2.1-48 
content  and  format,  2.1-52,  2.1-53 
further  definition  by  contractor,  2.1-53 
inclusion  in  Request  for  Proposal,  2.2-12,  2.2-13 
relation  to  other  contractual  documents,  2.1-52 
responsibilities  for  preparation,  2.1-49 
Statement  of  work,  sample  reliability  work  statements.  Appendix  C 
Statistical  confidence  levels,  for  reliability  requirements, 
2.1-22,  2.1-23 

Statistical  methods  research 
Air  Force  policy,  A. 1-2 
definition,  A* 1-2 
funding,  A. 1-4  ■ 
major  tasks,  A. 1-3 
Status  reporting 

Air  Force  internal  reports 
Air  Force  policy,  3*1-35 
definition,  3*1-35 
monthly  reliability  reports,  3*1-39 
quarterly  reliability  status  reports,  3*1-39 
responsibilities  for,  3*1-36 
sample  form  for  reliability  status  report,  3*1-41 
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Status  reporting  (continued) 
contractor  reports 

Air  Force  administrative  responsibilities,  3.1-4 

Air  Force  policy,  3*1-3 

check  list  for  report  evaluation,  3*1-15 

content  and  format  of  reports,  3. 1-5,  3*1-6,  A. 5-11, 

A. 5-12,  A.5-13 
definition,  3.1-3,  A. 5-10 
evaluation  of  reports  by  Air  Force,  3*1-7 
instructions  for  report  submittal,  3*1-5,  3*1-8 
schedule  for  report  submittal,  3*1-5,  3*1-6,  3*1-16 
A. 5-11,  A.5-13 

utilization  of  reports  by  Air  Force,  3*1-1^ 
Subcontractor  reliability  activities,  see  listings 
under  "vendor" 

Supplier  reliability  activities,  see  listings  under  "vendor" 
Surveillance  of  reliability  program,  see  program  surveillance 
Survey  questionnaire,  reliability.  Appendix  Z) 

System  definition,  A.  3-4 
System  effectiveness,  concept  of,  F-l 
System  engineering  process,  1-10 
Systems  management,  concept  of,  1-2 
System  mission,  description  of,  2.1-7,  A. 3-4 
System  parameters  affecting  reliability,  2.1-14,  F-l 
System  Program  Office  (SPO),  responsibilities  of 
Air  Force/contractor  liaison,  3*1-31 
contract  negotiation,  2.2-26 
contractor  data  submittals,  3.1-24 
coordination  with  AFPRO,  3*1-32,  3*1-34,  4.1-8 
enactment  of  reliability  program  plan,  2-3-3,  2.3-4 
preparation  of  reliability  requirements  statement,  2.1-49 
preparation  of  Request  for  Proposal,  2.2-9 
program  controls,  3*2-2 
program  review,  4.2-3 
program  surveillance,  4.1-2 
program  technical  direction,  3*3-2 
proposal  evaluation,  2.2-17 
reliability  status  reporting 

administration  of  contractor  reporting  activity,  3*1-4 
preparation  of  Air  Force  internal  reports ,  3*1-36 
selection  of  bidders,  2.2-2,  2.2-4 
selection  of  reliability  requirements,  2.1-5,  2.1-33 
system  management,  1-2 

Technical  direction  of  reliability  program,  see  program 
technical  direction 

Technical  meetings,  Air  Force/contractor 
conduct  of  meetings,  3. 1-20,  3. 1-21 
definition,  3*1-18 
pre-planning  activities,  3*1-19 
reporting  of  results,  3*1-20 
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Testing 

accelerated  tests,  A. 3-23 
acceptance  tests 

Air  Force  policy,  A.4-16 
definition,  A. 4-1 6 
developmental  tests 

Air  Porce  policy,  A. 3 -26 
definition,  A. 3 -26 

environmental  tests,  A. 3 -23,  A. 3-26,  A. 3-42 
field  tests,  A. 6-6,  A. 6-7 
parts  tests,  A. 1-13,  A. 1-17,  A. 3-19,  A. 3-22 
qualification  tests 
conduct  of,  A.4-16 
definition,  A.4-16 

reliability  demonstration  tests,  see  demonstration 
of  reliability  ~ 

Trade-offs 

in  design  reviews,  A.4-14 

reliability  and  other  system  parameters,  2.1-10 
use  of  trade-off  curves  in  program  direction,  3*3*5 
Training,  reliability 

for  contractor  field  representatives,  A. 2-9 
for  customer  representatives,  A. 2-9 
for  engineering  personnel,  A. 2-4,  A. 2-6 
for  management  personnel,  A. 2-4,  A. 2-6 
for  production  personnel,  A. 2-4,  A. 2-6 
for  support  personnel,  A. 2-5 
for  vendors,  A. 2-9 
formal  training 

Air  Fbrce  policy,  A. 2-3 
committee  for,  A. 2-5 
course  content,  A. 2-6,  A. 2-10 
definition,  A. 2-3 

indoctrination  in  reliability  requirements,  A. 2-4 
instructional  material,  A.  2-8,  A. 2-10 
Instructor  qualifications,  A. 2-8 
records,  A. 2-9 
scheduling,  A. 2-7,  A. 2-11 
motivational  training 

Air  Pbrce  policy,  A. 2-13 
definition,  A. 2-13 

materials  and  techniques  (films,  etc.),  3.4-7,  A. 2-13 
need  for,  A. 2-1 

Vendor  design  reviews,  A.4-14 
Vendor  parts  program,  A. 3-21 

Vendor  reliability  program  and  requirements,  A. 3-11,  A. 3-13, 
A. 3-21,  A.3-55,  A.3-56,  A.5-2 
Vendor  reliability  training,  A. 2-9 
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Vendor  selection  and  control 
Air  Force  policy,  A. 3 -53 

coordination  of  product  analysis  activities,  A. 3-58 
definition,  A. 3-53 

exemption  of  vendors  from  rating,  A. 3-57 
rating  of  vendors,  A. 3-55,  D-l 

surveillance  of  vendors  by  contractor,  A. 3-56,  A. 5-16, 
A. 5-17,  A. 5-18 

Weight  (system) /reliability  relationship,  2.1-17,  2.1-20 
Work  statement,  see  statement  of  reliability  requirements 
Sample  reliability  work  statements.  Appendix  C 
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AEROSPACE  CORPORATION 

foil  O  M  I  C I  SOX  •lots  •  10*  ANOIll*  4*.  CALIFORNIA 

OIIOINI  9-4*41 

1923. 1-64-84 
22  April  1964 

To:  All  Recipients  of  the  Reliability  Management  Handbook* 

1.  The  Reliability  Management  Handbook  is  designed  to  assist  Air  Force 
System  Program  Office  (SPO)  Reliability  Engineers  in  the  performance  of 
their  assigned  management  tasks.  This  Handbook  is  intended  to  serve  as 
an  aid  in  the  implementation  of  MIL-R-27542A  (USAF)  "Reliability  Program 
Requirements  for  Aerospace  Systems,  Subsystems,  and  Equipment." 


/ 


2.  It  is  recognized  that  certain  definitions  and  statements  of  philosophy 
expressed  herein  may  be  controversial.  However,  the  Handbook  is 
presented  as  an  expression  of  current  practice  in  the  field  of  reliability 
program  management.  Although  this  Handbook  has  been  extensively 
reviewed,  it  is  inevitable  that  not  all  of  the  errors  and  inconsistencies 
have  been  eliminated.  In  order  to  make  this  work  available  to  you  within 
a  reasonable  time,  while  it  is  still  meaningful,  it  is  published  in  its 
present  imperfect  form.  It  is  the  intention  of  the  Aerospace  Corporation 
to  review  periodically  and  update  the  Handbook  as  required. 

3.  Recommendations  for  improvement  of  the  Handbook  are  solicited 
from  users,  interested  individuals,  and  organizations.  It  iB  recommended 
that  appropriate  comments  be  made  directly  on  the  pages  of  the  Handbook. 
Copies  of  the  marked  pages  and  comments  should  be  forwarded  to  the 
attention  of  F.  P.  Klein,  Reliability  Department,  Aerospace  Corporation, 
or  Lt.  Col.  O.  A.  Bernhoff,  SSD  (SSSIR),  USAF. 

4.  The  Handbook  is  constructed  so  that  individual  pages  or  sections  may 
be  revised  as  required.  It  is  intended  that  recipients  of  the  Handbook 
receive  copies  of  each  revision.  ,*  It  is  requested  that  each  addressee 
sign  the  attached  receipt  and  forward  it  to  this  office  in  order  to  ensure 
^he  receipt  of  future  corrections,  changes,  and  additions  to  this  Handbook. 


\ 
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Very  truly  yours. 


F.  P.  Klein 
Head,  Reliability  Department 


PK/gm 

Attch.(l)  (To  be  completed  by  the  recipient) 

*  Reliability  Management  Handbook,  TOR-269(4303)-9,  Vol.  I  and  II 
Replaces  Previously  Published  Handbook  ATN- 64(4303)- 1,  dated 
3  September  1963,  Vol.  I  and  Q. 
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1923. 1-64.-84 
Attachment  1 


F.  P.  Klein 

Head,  Reliability  Department 
Aerospace  Corporation 
P.  O.  Box  95085 
Los  Angeles  45,  California 


The  Reliability  Management  Handbook,  Volumes  I  and  II 
were  received  by  my  office  and  are  denoted  by  Serial 
Number  I/O  As  corrections,  changes,  and  additions 
are  made  to  the  Handbook,  I  will  insert  same  into  the  Hand¬ 
book  as  noted. 

Name:  (Please  Print) _ 

Signature:  _ Date: _ 

Address: _ Bldg. _ Room _ 

City: _ _ State : _ 

Recipient's  Organization  Code _ . 


